[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:40 dist-test-slave-gkhx krb5kdc[14212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:40 dist-test-slave-gkhx krb5kdc[14212](info): set up 2 sockets
Sep 28 14:45:40 dist-test-slave-gkhx krb5kdc[14212](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 14:45:41.097111 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:41.248787 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37823
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:37823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:41.375049 14225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:41.375418 14225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:41.375494 14225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:41.379928 14225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:41.380044 14225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:41.380084 14225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:41.380122 14225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:41.380159 14225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:41.386353 14225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:37823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:41.387765 14225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:41.388981 14225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:41.399644 14225 server_base.cc:1034] running on GCE node
W20260928 14:45:41.399537 14233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:41.399673 14231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:41.399544 14230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:41.400677 14225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:41.402127 14225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:41.403352 14225 hybrid_clock.cc:648] HybridClock initialized: now 1790606741403318 us; error 55 us; skew 500 ppm
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:41.406641 14225 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:41.407761 14225 webserver.cc:460] Webserver started at http://127.13.223.126:34925/ using document root <none> and password file <none>
I20260928 14:45:41.408390 14225 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:41.408456 14225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:41.408711 14225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:41.411221 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "f780938e40aa47dfabc6e6bdd75243bb"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "0238c5bb142f008828b2b6ae8a9e4057"
server_key_iv: "58d3026381d4208825ed609de338b230"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.411945 14225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "f780938e40aa47dfabc6e6bdd75243bb"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "0238c5bb142f008828b2b6ae8a9e4057"
server_key_iv: "58d3026381d4208825ed609de338b230"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.416915 14225 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:45:41.420235 14240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:41.421993 14225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:41.422260 14225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "f780938e40aa47dfabc6e6bdd75243bb"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "0238c5bb142f008828b2b6ae8a9e4057"
server_key_iv: "58d3026381d4208825ed609de338b230"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.422412 14225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:41.447942 14225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:41.451965 14225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:41.452303 14225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:41.461113 14225 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:37823
I20260928 14:45:41.461236 14293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:37823 every 8 connection(s)
I20260928 14:45:41.462293 14225 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:41.465723 14294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:41.468124 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14225
I20260928 14:45:41.468293 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:41.469843 14205 external_mini_cluster.cc:1428] Setting key 2812ef913e052aa202989c84a0b46a7d
I20260928 14:45:41.472293 14294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb: Bootstrap starting.
I20260928 14:45:41.475087 14294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:41.475988 14294 log.cc:824] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:41.478263 14294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb: No bootstrap required, opened a new log
I20260928 14:45:41.482071 14294 raft_consensus.cc:348] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } }
I20260928 14:45:41.482380 14294 raft_consensus.cc:374] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:41.482467 14294 raft_consensus.cc:729] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f780938e40aa47dfabc6e6bdd75243bb, State: Initialized, Role: FOLLOWER
I20260928 14:45:41.482967 14294 consensus_queue.cc:260] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } }
I20260928 14:45:41.483132 14294 raft_consensus.cc:388] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:41.483217 14294 raft_consensus.cc:482] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:41.483335 14294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:41.485397 14294 raft_consensus.cc:504] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } }
I20260928 14:45:41.485814 14294 leader_election.cc:302] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f780938e40aa47dfabc6e6bdd75243bb; no voters:
I20260928 14:45:41.486249 14294 leader_election.cc:288] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:41.486343 14299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:41.486611 14299 raft_consensus.cc:686] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [term 1 LEADER]: Becoming Leader. State: Replica: f780938e40aa47dfabc6e6bdd75243bb, State: Running, Role: LEADER
I20260928 14:45:41.486965 14299 consensus_queue.cc:237] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } }
I20260928 14:45:41.487890 14294 sys_catalog.cc:564] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:41.489013 14301 sys_catalog.cc:455] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader f780938e40aa47dfabc6e6bdd75243bb. Latest consensus state: current_term: 1 leader_uuid: "f780938e40aa47dfabc6e6bdd75243bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } } }
I20260928 14:45:41.489192 14301 sys_catalog.cc:458] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:41.489005 14300 sys_catalog.cc:455] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f780938e40aa47dfabc6e6bdd75243bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f780938e40aa47dfabc6e6bdd75243bb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37823 } } }
I20260928 14:45:41.489451 14300 sys_catalog.cc:458] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:41.489768 14303 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:41.493381 14303 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:41.500190 14303 catalog_manager.cc:1347] Generated new cluster ID: 86a2db78d9224a55b3548a510a74821b
I20260928 14:45:41.500280 14303 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:41.501423 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:41.473528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38346 (local address 127.13.223.126:37823)
0928 14:45:41.474125 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:41.474158 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:41.475551 (+ 1393us) server_negotiation.cc:407] Connection header received
0928 14:45:41.479276 (+ 3725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:41.479297 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:41.479689 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:41.480120 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:41.482995 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.484506 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.486138 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.486981 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.487923 (+ 942us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:41.493589 (+ 5666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:41.493601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:41.493624 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:41.493676 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:41.495976 (+ 2300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:41.496728 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:41.496734 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:41.496739 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:41.496939 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:41.497408 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:41.497415 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:41.497417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:41.498086 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:41.498323 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:41.499453 (+ 1130us) server_negotiation.cc:299] Negotiation successful
0928 14:45:41.499828 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":189,"threads_started":1}
I20260928 14:45:41.515131 14303 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:41.516690 14303 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:41.526276 14303 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f780938e40aa47dfabc6e6bdd75243bb: Generated new TSK 0
I20260928 14:45:41.527333 14303 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:45:41.572727 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:41.697507 14320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:41.697916 14320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:41.697999 14320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:41.702605 14320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:41.702716 14320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:41.702811 14320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:45:41.709981 14320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:41.711328 14320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:41.712518 14320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:41.724205 14328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:41.724205 14326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:41.724484 14320 server_base.cc:1034] running on GCE node
W20260928 14:45:41.724208 14325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:41.725049 14320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:41.725787 14320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:41.726977 14320 hybrid_clock.cc:648] HybridClock initialized: now 1790606741726961 us; error 34 us; skew 500 ppm
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:41.730739 14320 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:45:41.731969 14320 webserver.cc:460] Webserver started at http://127.13.223.65:35409/ using document root <none> and password file <none>
I20260928 14:45:41.732674 14320 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:41.732779 14320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:41.733049 14320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:41.735641 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "8579948f72364e0786cdf0f308592488"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "e82fe21ad033e868225d884931d5d6c6"
server_key_iv: "d414fe08225fc8480630c0e4b561394c"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.736371 14320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "8579948f72364e0786cdf0f308592488"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "e82fe21ad033e868225d884931d5d6c6"
server_key_iv: "d414fe08225fc8480630c0e4b561394c"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.741379 14320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:45:41.744628 14335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:41.746246 14320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:41.746425 14320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "8579948f72364e0786cdf0f308592488"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "e82fe21ad033e868225d884931d5d6c6"
server_key_iv: "d414fe08225fc8480630c0e4b561394c"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.746603 14320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:41.761561 14320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:41.765718 14320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:41.765987 14320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:41.766758 14320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:41.767789 14320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:41.767868 14320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:41.767947 14320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:41.768003 14320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:41.779641 14320 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:34823
I20260928 14:45:41.779687 14448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:34823 every 8 connection(s)
I20260928 14:45:41.780759 14320 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:45:41.781241 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14320
I20260928 14:45:41.781379 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:45:41.781672 14205 external_mini_cluster.cc:1428] Setting key c205c830fa19c2420877a2631bfffcec
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:41.801555 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:41.783006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:38079 (local address 127.13.223.126:37823)
0928 14:45:41.783187 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:41.783194 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:41.784741 (+ 1547us) server_negotiation.cc:407] Connection header received
0928 14:45:41.785869 (+ 1128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:41.785872 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:41.785939 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:41.786054 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:41.788026 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.789038 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.790985 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.791415 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.791562 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:41.797129 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:41.797134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:41.797137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:41.797184 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:41.798841 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:41.799685 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:41.799689 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:41.799690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:41.799791 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:41.800263 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:41.800267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:41.800268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:41.800488 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:41.800599 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:41.801220 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 14:45:41.801379 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:45:41.802671 14451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:41.783571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37823 (local address 127.13.223.65:38079)
0928 14:45:41.784502 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:41.784573 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:41.785611 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:41.786444 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:41.786453 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:41.786989 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:41.787803 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:41.787831 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.789240 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.789243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:41.790755 (+ 1512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:41.790770 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:41.791658 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:41.791662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:41.791864 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:41.792791 (+ 927us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:41.792812 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:41.796868 (+ 4056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:41.799028 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:41.799035 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:41.799062 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:41.799420 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:41.799917 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:41.799921 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:41.799923 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:41.800132 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:41.800620 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:41.800629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:41.800944 (+ 315us) client_negotiation.cc:769] Sending connection context
0928 14:45:41.801157 (+ 213us) client_negotiation.cc:240] Negotiation successful
0928 14:45:41.801576 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":146,"threads_started":1}
I20260928 14:45:41.804394 14449 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37823
I20260928 14:45:41.804716 14449 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:41.805414 14449 heartbeater.cc:511] Master 127.13.223.126:37823 requested a full tablet report, sending...
I20260928 14:45:41.807376 14258 ts_manager.cc:194] Registered new tserver with Master: 8579948f72364e0786cdf0f308592488 (127.13.223.65:34823)
I20260928 14:45:41.809171 14258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:38079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:45:41.818255 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:41.946184 14454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:41.946555 14454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:41.946693 14454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:41.951133 14454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:41.951247 14454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:41.951346 14454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:45:41.957320 14454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:41.958730 14454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:41.959929 14454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:41.971822 14459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:41.971836 14462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:41.971916 14460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:41.971906 14454 server_base.cc:1034] running on GCE node
I20260928 14:45:41.972653 14454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:41.973423 14454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:41.974663 14454 hybrid_clock.cc:648] HybridClock initialized: now 1790606741974633 us; error 39 us; skew 500 ppm
Sep 28 14:45:41 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:41.977820 14454 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:45:41.979015 14454 webserver.cc:460] Webserver started at http://127.13.223.66:37739/ using document root <none> and password file <none>
I20260928 14:45:41.979610 14454 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:41.979674 14454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:41.979852 14454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:41.982085 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "d70b84cf813b4295a64bf52b98546a69"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "fc174d7ab1cdcca7e3c418999cabc57c"
server_key_iv: "7a0edbe2731dc0250fc07a5d1f936c36"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.982776 14454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "d70b84cf813b4295a64bf52b98546a69"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "fc174d7ab1cdcca7e3c418999cabc57c"
server_key_iv: "7a0edbe2731dc0250fc07a5d1f936c36"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.987080 14454 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:45:41.990113 14469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:41.991882 14454 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:41.992013 14454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "d70b84cf813b4295a64bf52b98546a69"
format_stamp: "Formatted at 2026-09-28 14:45:41 on dist-test-slave-gkhx"
server_key: "fc174d7ab1cdcca7e3c418999cabc57c"
server_key_iv: "7a0edbe2731dc0250fc07a5d1f936c36"
server_key_version: "encryptionkey@0"
I20260928 14:45:41.992136 14454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:42.014159 14454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:42.017978 14454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:42.018188 14454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:42.018879 14454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:42.019748 14454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:42.019797 14454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:42.019841 14454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:42.019857 14454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:42.030498 14454 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:45107
I20260928 14:45:42.030517 14582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:45107 every 8 connection(s)
I20260928 14:45:42.031623 14454 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:45:42.036442 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14454
I20260928 14:45:42.036593 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:45:42.036840 14205 external_mini_cluster.cc:1428] Setting key d63d67509be7e68dc9ee32b3b681ef56
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606741, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:45:42.048511 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.033924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:50869 (local address 127.13.223.126:37823)
0928 14:45:42.034189 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:42.034195 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:42.035395 (+ 1200us) server_negotiation.cc:407] Connection header received
0928 14:45:42.036416 (+ 1021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:42.036420 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:42.036475 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:42.036610 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:42.038385 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.039016 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.040334 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.040570 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.040690 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.044131 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:42.044136 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:42.044139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:42.044176 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:42.045966 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.046688 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.046692 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.046698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.046785 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.047193 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.047196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.047198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.047451 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:42.047559 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:42.048188 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 14:45:42.048333 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:45:42.049516 14585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.034359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37823 (local address 127.13.223.66:50869)
0928 14:45:42.035209 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:42.035259 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:42.036129 (+ 870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:42.036833 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:42.036842 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:42.037386 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:42.038220 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:42.038243 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.039175 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.039178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:42.040186 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:42.040197 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.040749 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.040754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:42.040958 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.041766 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:42.041786 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:42.043856 (+ 2070us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:42.046148 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:42.046156 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:42.046181 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:42.046517 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:42.046888 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:42.046891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:42.046893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:42.047080 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:42.047560 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:42.047571 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:42.047882 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 14:45:42.048094 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 14:45:42.048495 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":206,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:45:42.051538 14583 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37823
I20260928 14:45:42.051874 14583 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:42.052573 14583 heartbeater.cc:511] Master 127.13.223.126:37823 requested a full tablet report, sending...
I20260928 14:45:42.054098 14258 ts_manager.cc:194] Registered new tserver with Master: d70b84cf813b4295a64bf52b98546a69 (127.13.223.66:45107)
I20260928 14:45:42.055078 14258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:50869
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:45:42.071610 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:42.197189 14588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:42.197541 14588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:42.197616 14588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:42.202078 14588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:42.202205 14588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:42.202306 14588 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:45:42.209192 14588 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37823
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14588
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:42.210433 14588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:42.211611 14588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:42.222965 14596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:42.222975 14593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:42.223028 14594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:42.223242 14588 server_base.cc:1034] running on GCE node
I20260928 14:45:42.223850 14588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:42.224795 14588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:42.226038 14588 hybrid_clock.cc:648] HybridClock initialized: now 1790606742226012 us; error 42 us; skew 500 ppm
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:42.229887 14588 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:45:42.231058 14588 webserver.cc:460] Webserver started at http://127.13.223.67:46207/ using document root <none> and password file <none>
I20260928 14:45:42.231786 14588 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:42.231859 14588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:42.232070 14588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:42.234716 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "83f4254843bb47d09d217f66e8a77ba9"
format_stamp: "Formatted at 2026-09-28 14:45:42 on dist-test-slave-gkhx"
server_key: "a78b8de37da1148faa6be92061364c5d"
server_key_iv: "8a45addcd54a61ce2a2b391ec836dd8f"
server_key_version: "encryptionkey@0"
I20260928 14:45:42.235597 14588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "83f4254843bb47d09d217f66e8a77ba9"
format_stamp: "Formatted at 2026-09-28 14:45:42 on dist-test-slave-gkhx"
server_key: "a78b8de37da1148faa6be92061364c5d"
server_key_iv: "8a45addcd54a61ce2a2b391ec836dd8f"
server_key_version: "encryptionkey@0"
I20260928 14:45:42.240656 14588 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:45:42.244336 14603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:42.246120 14588 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:45:42.246335 14588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "83f4254843bb47d09d217f66e8a77ba9"
format_stamp: "Formatted at 2026-09-28 14:45:42 on dist-test-slave-gkhx"
server_key: "a78b8de37da1148faa6be92061364c5d"
server_key_iv: "8a45addcd54a61ce2a2b391ec836dd8f"
server_key_version: "encryptionkey@0"
I20260928 14:45:42.246496 14588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:42.261560 14588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:42.265652 14588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:42.265919 14588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:42.266752 14588 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:42.268224 14588 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:42.268297 14588 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:42.268368 14588 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:42.268404 14588 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:42.281877 14588 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:33657
I20260928 14:45:42.281895 14716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:33657 every 8 connection(s)
I20260928 14:45:42.283125 14588 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:45:42.289695 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14588
I20260928 14:45:42.289839 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:45:42.290210 14205 external_mini_cluster.cc:1428] Setting key 8da1a7c9578b3ea58041c30a4b1c6677
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:42.301400 14296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.285413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44367 (local address 127.13.223.126:37823)
0928 14:45:42.285576 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:42.285581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:42.286773 (+ 1192us) server_negotiation.cc:407] Connection header received
0928 14:45:42.288292 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:42.288295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:42.288359 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:42.288466 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:42.290200 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.291107 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.292967 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.293270 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.293403 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.296743 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:42.296747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:42.296750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:42.296784 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:42.298655 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.299367 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.299371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.299372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.299474 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.299969 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.299972 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.299974 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.300248 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:42.300345 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:42.301058 (+ 713us) server_negotiation.cc:299] Negotiation successful
0928 14:45:42.301256 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:45:42.302517 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.285920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37823 (local address 127.13.223.67:44367)
0928 14:45:42.286526 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:42.286653 (+ 127us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:42.288023 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:42.288675 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:42.288684 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:42.289170 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:42.289904 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:42.289929 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.291319 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.291322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:42.292806 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:42.292820 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.293460 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.293464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:42.293692 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.294373 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:42.294393 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:42.296317 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:42.298835 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:42.298843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:42.298861 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:42.299240 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:42.299613 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:42.299617 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:42.299619 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:42.299841 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:42.300389 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:42.300396 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:42.300761 (+ 365us) client_negotiation.cc:769] Sending connection context
0928 14:45:42.300977 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:45:42.301342 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":154,"threads_started":1}
I20260928 14:45:42.304275 14717 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37823
I20260928 14:45:42.304610 14717 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:42.305315 14717 heartbeater.cc:511] Master 127.13.223.126:37823 requested a full tablet report, sending...
I20260928 14:45:42.306731 14258 ts_manager.cc:194] Registered new tserver with Master: 83f4254843bb47d09d217f66e8a77ba9 (127.13.223.67:33657)
I20260928 14:45:42.307492 14258 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:44367
I20260928 14:45:42.318866 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:42.349143 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.337310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34596 (local address 127.13.223.65:34823)
0928 14:45:42.337650 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:42.337664 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:42.337691 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:45:42.337806 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:42.337812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:42.338019 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:42.338164 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:42.339086 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.339927 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.340879 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.341313 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.341407 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.344644 (+ 3237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:42.344660 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:42.344676 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:42.344712 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:42.346869 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.347469 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.347476 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.347477 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.347590 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.347925 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.347929 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.347931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.348258 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:42.348535 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:42.348738 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:45:42.348925 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":143,"threads_started":1}
W20260928 14:45:42.350293 14363 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34596
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14212](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:42.382169 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:42.371322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34612 (local address 127.13.223.65:34823)
0928 14:45:42.371552 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:42.371558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:42.371731 (+ 173us) server_negotiation.cc:407] Connection header received
0928 14:45:42.371934 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:42.371938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:42.371987 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:42.372091 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:42.373157 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.373716 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.374693 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:42.374950 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:42.375072 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:42.378163 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:42.378167 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:42.378169 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:42.378204 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:42.380085 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.380653 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.380656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.380658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.380755 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:42.381153 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:42.381157 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:42.381159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:42.381379 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:42.381551 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:42.381829 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:45:42.381978 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 14:45:42.383808 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14320
I20260928 14:45:42.390895 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14454
I20260928 14:45:42.399804 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14588
I20260928 14:45:42.406289 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14225
2026-09-28T14:45:42Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2409 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14731](info): set up 2 sockets
Sep 28 14:45:42 dist-test-slave-gkhx krb5kdc[14731](info): commencing operation
krb5kdc: starting...
W20260928 14:45:43.488453 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:43 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:43.619043 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42925
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:42925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:43.735616 14744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:43.735949 14744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:43.735994 14744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:43.740347 14744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:43.740437 14744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:43.740463 14744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:43.740481 14744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:43.740497 14744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:43.746342 14744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:42925
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:43.747613 14744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:43.748678 14744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:43.758051 14744 server_base.cc:1034] running on GCE node
W20260928 14:45:43.757974 14752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:43.758001 14749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:43.757974 14750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:43.759037 14744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:43.760326 14744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:43.761540 14744 hybrid_clock.cc:648] HybridClock initialized: now 1790606743761520 us; error 38 us; skew 500 ppm
Sep 28 14:45:43 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606743, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:43.771728 14744 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:43.772921 14744 webserver.cc:460] Webserver started at http://127.13.223.126:44227/ using document root <none> and password file <none>
I20260928 14:45:43.773593 14744 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:43.773693 14744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:43.773937 14744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:43.776237 14744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "708a65d43bd24457a4f16a4429331459"
format_stamp: "Formatted at 2026-09-28 14:45:43 on dist-test-slave-gkhx"
server_key: "1ff32885514a32cfea506b2380354e38"
server_key_iv: "af1efc5111c9cacaccb4e84f7ebabfc4"
server_key_version: "encryptionkey@0"
I20260928 14:45:43.776898 14744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "708a65d43bd24457a4f16a4429331459"
format_stamp: "Formatted at 2026-09-28 14:45:43 on dist-test-slave-gkhx"
server_key: "1ff32885514a32cfea506b2380354e38"
server_key_iv: "af1efc5111c9cacaccb4e84f7ebabfc4"
server_key_version: "encryptionkey@0"
I20260928 14:45:43.781136 14744 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:45:43.784011 14759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:43.785449 14744 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:45:43.785629 14744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "708a65d43bd24457a4f16a4429331459"
format_stamp: "Formatted at 2026-09-28 14:45:43 on dist-test-slave-gkhx"
server_key: "1ff32885514a32cfea506b2380354e38"
server_key_iv: "af1efc5111c9cacaccb4e84f7ebabfc4"
server_key_version: "encryptionkey@0"
I20260928 14:45:43.785768 14744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:43.804540 14744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:43.808300 14744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:43.808537 14744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:43.816414 14744 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:42925
I20260928 14:45:43.816454 14811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:42925 every 8 connection(s)
I20260928 14:45:43.817525 14744 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:43.820824 14812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:43.826463 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14744
I20260928 14:45:43.826627 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:43.826877 14205 external_mini_cluster.cc:1428] Setting key 35d902af7b6018e5c07a4109aa1f6412
I20260928 14:45:43.827477 14812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459: Bootstrap starting.
I20260928 14:45:43.830376 14812 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:43.831560 14812 log.cc:824] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:43.834232 14812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459: No bootstrap required, opened a new log
Sep 28 14:45:43 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:43.839588 14812 raft_consensus.cc:348] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } }
I20260928 14:45:43.839923 14812 raft_consensus.cc:374] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:43.840014 14812 raft_consensus.cc:729] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 708a65d43bd24457a4f16a4429331459, State: Initialized, Role: FOLLOWER
I20260928 14:45:43.840579 14812 consensus_queue.cc:260] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } }
I20260928 14:45:43.840755 14812 raft_consensus.cc:388] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:43.840847 14812 raft_consensus.cc:482] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:43.840987 14812 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:43.842089 14812 raft_consensus.cc:504] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } }
I20260928 14:45:43.842586 14812 leader_election.cc:302] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 708a65d43bd24457a4f16a4429331459; no voters:
I20260928 14:45:43.842999 14812 leader_election.cc:288] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:43.843132 14817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:43.843384 14817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [term 1 LEADER]: Becoming Leader. State: Replica: 708a65d43bd24457a4f16a4429331459, State: Running, Role: LEADER
I20260928 14:45:43.843778 14817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } }
I20260928 14:45:43.844177 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:43.828342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.13.223.126:42925)
0928 14:45:43.828805 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:43.828822 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:43.828862 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:45:43.829566 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:43.829586 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:43.830023 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:43.830337 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:43.831529 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:43.832767 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:43.834103 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:43.834725 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:43.834854 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:43.838007 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:43.838013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:43.838030 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:43.838061 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:43.840163 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:43.840716 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:43.840728 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:43.840735 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:43.840928 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:43.841277 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:43.841282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:43.841284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:43.841807 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:43.842060 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:43.842379 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 14:45:43.842781 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":167,"threads_started":1}
I20260928 14:45:43.844632 14812 sys_catalog.cc:564] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:43.845204 14817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 708a65d43bd24457a4f16a4429331459. Latest consensus state: current_term: 1 leader_uuid: "708a65d43bd24457a4f16a4429331459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } } }
I20260928 14:45:43.845328 14817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:43.845671 14818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "708a65d43bd24457a4f16a4429331459" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "708a65d43bd24457a4f16a4429331459" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42925 } } }
I20260928 14:45:43.845788 14818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:43.846903 14821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:43.850718 14821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:43.858119 14821 catalog_manager.cc:1347] Generated new cluster ID: 6076fe0ad6d746ebb097f621e1271a4d
I20260928 14:45:43.858250 14821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:43.876039 14821 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:43.877429 14821 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:43.881556 14821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 708a65d43bd24457a4f16a4429331459: Generated new TSK 0
I20260928 14:45:43.882323 14821 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:45:43.920109 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:44.049851 14838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:44.050168 14838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:44.050215 14838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:44.054529 14838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:44.054646 14838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:44.054754 14838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:45:44.060412 14838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:44.061626 14838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:44.062788 14838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:44.073207 14844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:44.073276 14846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:44.073223 14843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:44.073686 14838 server_base.cc:1034] running on GCE node
I20260928 14:45:44.074187 14838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:44.075029 14838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:44.076258 14838 hybrid_clock.cc:648] HybridClock initialized: now 1790606744076251 us; error 36 us; skew 500 ppm
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:44.079593 14838 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:45:44.080631 14838 webserver.cc:460] Webserver started at http://127.13.223.65:42385/ using document root <none> and password file <none>
I20260928 14:45:44.081319 14838 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:44.081414 14838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:44.081669 14838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:44.083971 14838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "f9d7f060e8134464b5cfc1376ea5cace"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2c9a4d99cc28fa89f4584deb3ab70929"
server_key_iv: "b0292b61c390dc5a37a4db9537f3017b"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.084615 14838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "f9d7f060e8134464b5cfc1376ea5cace"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2c9a4d99cc28fa89f4584deb3ab70929"
server_key_iv: "b0292b61c390dc5a37a4db9537f3017b"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.089535 14838 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:45:44.092645 14853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.094240 14838 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:44.094419 14838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "f9d7f060e8134464b5cfc1376ea5cace"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2c9a4d99cc28fa89f4584deb3ab70929"
server_key_iv: "b0292b61c390dc5a37a4db9537f3017b"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.094590 14838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:44.117774 14838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:44.122150 14838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:44.122455 14838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:44.123304 14838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:44.124337 14838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:44.124428 14838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.124509 14838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:44.124562 14838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.134989 14838 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:41399
I20260928 14:45:44.135008 14966 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:41399 every 8 connection(s)
I20260928 14:45:44.136047 14838 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:45:44.138470 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14838
I20260928 14:45:44.138602 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:45:44.138892 14205 external_mini_cluster.cc:1428] Setting key 06b067b3e602d0a3de7267c1109d2303
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:45:44.151618 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.138133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:53655 (local address 127.13.223.126:42925)
0928 14:45:44.138321 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.138326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.139468 (+ 1142us) server_negotiation.cc:407] Connection header received
0928 14:45:44.140577 (+ 1109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.140581 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.140636 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.140708 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:44.141964 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.142501 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.143765 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.144008 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.144122 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.147320 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:44.147325 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:44.147328 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:44.147367 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:44.149083 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.149723 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.149727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.149728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.149824 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.150228 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.150234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.150236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.150465 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:44.150610 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.151240 (+ 630us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.151409 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:45:44.152565 14969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.138700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42925 (local address 127.13.223.65:53655)
0928 14:45:44.139293 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.139341 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.140368 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.140887 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.140894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.141279 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:44.141806 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.141822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.142685 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.142689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.143634 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.143644 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.144178 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.144182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.144384 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.145127 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:44.145145 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:44.147104 (+ 1959us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:44.149238 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.149245 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.149265 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.149613 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.149938 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.149942 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.149944 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.150120 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.150616 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:44.150626 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:44.150951 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.151195 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.151596 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":151,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:45:44.154268 14967 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42925
I20260928 14:45:44.154621 14967 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:44.155306 14967 heartbeater.cc:511] Master 127.13.223.126:42925 requested a full tablet report, sending...
I20260928 14:45:44.157083 14776 ts_manager.cc:194] Registered new tserver with Master: f9d7f060e8134464b5cfc1376ea5cace (127.13.223.65:41399)
I20260928 14:45:44.158386 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:53655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:45:44.170415 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:44.294633 14972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:44.295040 14972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:44.295099 14972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:44.299587 14972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:44.299664 14972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:44.299731 14972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:45:44.305464 14972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:44.306663 14972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:44.307840 14972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:44.318982 14972 server_base.cc:1034] running on GCE node
W20260928 14:45:44.318912 14977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:44.318934 14978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:44.318912 14980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:44.319746 14972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:44.320539 14972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:44.321748 14972 hybrid_clock.cc:648] HybridClock initialized: now 1790606744321732 us; error 30 us; skew 500 ppm
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:44.325193 14972 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:45:44.326159 14972 webserver.cc:460] Webserver started at http://127.13.223.66:41469/ using document root <none> and password file <none>
I20260928 14:45:44.326838 14972 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:44.326907 14972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:44.327154 14972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:44.329624 14972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "4e3cc95f05da47218bca5da015af2852"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2cfd22bd82a5c8f684e994553290ba77"
server_key_iv: "f7aa3bb572e6d5807ef0bada46b9cff4"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.330238 14972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "4e3cc95f05da47218bca5da015af2852"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2cfd22bd82a5c8f684e994553290ba77"
server_key_iv: "f7aa3bb572e6d5807ef0bada46b9cff4"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.334789 14972 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 14:45:44.337770 14987 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.339359 14972 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:44.339529 14972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "4e3cc95f05da47218bca5da015af2852"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "2cfd22bd82a5c8f684e994553290ba77"
server_key_iv: "f7aa3bb572e6d5807ef0bada46b9cff4"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.339668 14972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:44.361920 14972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:44.365679 14972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:44.365929 14972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:44.366695 14972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:44.367705 14972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:44.367775 14972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.367853 14972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:44.367895 14972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.378261 14972 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:34651
I20260928 14:45:44.378284 15100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:34651 every 8 connection(s)
I20260928 14:45:44.379385 14972 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:45:44.388173 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 14972
I20260928 14:45:44.388314 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:45:44.388674 14205 external_mini_cluster.cc:1428] Setting key 06d70897a88fe2dcaec3be7f18ba905d
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:44.395824 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.381513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:37361 (local address 127.13.223.126:42925)
0928 14:45:44.381684 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.381688 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.382666 (+ 978us) server_negotiation.cc:407] Connection header received
0928 14:45:44.383697 (+ 1031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.383700 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.383762 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.383846 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:44.385308 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.385880 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.387300 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.387560 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.387661 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.390944 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:44.390948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:44.390950 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:44.390977 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:44.392987 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.393702 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.393709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.393715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.393823 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.394283 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.394289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.394294 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.394622 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:44.394738 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.395428 (+ 690us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.395620 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:45:44.396703 15103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.381981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42925 (local address 127.13.223.66:37361)
0928 14:45:44.382474 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.382523 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.383443 (+ 920us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.384115 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.384123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.384550 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:44.385131 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.385148 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.386087 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.386090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.387179 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.387190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.387717 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.387720 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.387870 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.388652 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:44.388676 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:44.390747 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:44.393137 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.393143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.393159 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.393517 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.393956 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.393959 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.393960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.394143 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.394767 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:44.394773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:44.395052 (+ 279us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.395325 (+ 273us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.395666 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":119,"threads_started":1}
I20260928 14:45:44.398021 15101 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42925
I20260928 14:45:44.398409 15101 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:44.399063 15101 heartbeater.cc:511] Master 127.13.223.126:42925 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:45:44.400308 14776 ts_manager.cc:194] Registered new tserver with Master: 4e3cc95f05da47218bca5da015af2852 (127.13.223.66:34651)
I20260928 14:45:44.401074 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:37361
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:45:44.422856 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:44.542599 15106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:44.542930 15106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:44.542979 15106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:44.547616 15106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:44.547696 15106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:44.547762 15106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:45:44.553498 15106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42925
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:44.554759 15106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:44.555884 15106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:44.566692 15111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:44.566839 15106 server_base.cc:1034] running on GCE node
W20260928 14:45:44.566715 15112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:44.567047 15114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:44.567538 15106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:44.568331 15106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:44.569523 15106 hybrid_clock.cc:648] HybridClock initialized: now 1790606744569509 us; error 30 us; skew 500 ppm
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:44.572989 15106 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:45:44.574164 15106 webserver.cc:460] Webserver started at http://127.13.223.67:45539/ using document root <none> and password file <none>
I20260928 14:45:44.574826 15106 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:44.574898 15106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:44.575114 15106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:44.577245 15106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "5f3209d20e8a4836a4da5602779d93e9"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "379493e0254661306c70f5ce790b98a5"
server_key_iv: "bf478746c39303d85a17b31e51e4d5ab"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.577831 15106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "5f3209d20e8a4836a4da5602779d93e9"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "379493e0254661306c70f5ce790b98a5"
server_key_iv: "bf478746c39303d85a17b31e51e4d5ab"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.581856 15106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:45:44.584846 15121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.586613 15106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:45:44.586781 15106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "5f3209d20e8a4836a4da5602779d93e9"
format_stamp: "Formatted at 2026-09-28 14:45:44 on dist-test-slave-gkhx"
server_key: "379493e0254661306c70f5ce790b98a5"
server_key_iv: "bf478746c39303d85a17b31e51e4d5ab"
server_key_version: "encryptionkey@0"
I20260928 14:45:44.586956 15106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:44.609272 15106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:44.613163 15106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:44.613401 15106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:44.614053 15106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:44.614992 15106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:44.615063 15106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.615170 15106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:44.615216 15106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:44.625324 15106 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:38269
I20260928 14:45:44.625372 15234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:38269 every 8 connection(s)
I20260928 14:45:44.626377 15106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:45:44.630759 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15106
I20260928 14:45:44.630911 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:45:44.631224 14205 external_mini_cluster.cc:1428] Setting key 1dbeb9ca0f6c4b1a465adfe45321b28f
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606744, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:44.642899 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.628508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:47269 (local address 127.13.223.126:42925)
0928 14:45:44.628698 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.628702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.629580 (+ 878us) server_negotiation.cc:407] Connection header received
0928 14:45:44.631434 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.631437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.631491 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.631597 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:44.633246 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.633874 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.635288 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.635560 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.635664 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.638669 (+ 3005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:44.638673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:44.638675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:44.638703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:44.640404 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.641054 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.641058 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.641059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.641153 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.641591 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.641595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.641596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.641833 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:44.641922 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.642636 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.642761 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 14:45:44.643930 15237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.628931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42925 (local address 127.13.223.67:47269)
0928 14:45:44.629409 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.629460 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.631176 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.631795 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.631805 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.632401 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:44.633103 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.633119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.634015 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.634018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.635142 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.635153 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.635757 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.635760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.635988 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.636702 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:44.636720 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:44.638420 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:44.640565 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.640572 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.640590 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.640889 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.641296 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:44.641299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:44.641300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:44.641474 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:44.641943 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:44.641949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:44.642308 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.642496 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.642918 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":119,"threads_started":1}
I20260928 14:45:44.645378 15235 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42925
I20260928 14:45:44.645736 15235 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:44.646371 15235 heartbeater.cc:511] Master 127.13.223.126:42925 requested a full tablet report, sending...
I20260928 14:45:44.647643 14776 ts_manager.cc:194] Registered new tserver with Master: 5f3209d20e8a4836a4da5602779d93e9 (127.13.223.67:38269)
I20260928 14:45:44.648432 14776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:47269
I20260928 14:45:44.658094 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:45:44.669857 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.662070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47752 (local address 127.13.223.126:42925)
0928 14:45:44.662282 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.662287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.662433 (+ 146us) server_negotiation.cc:407] Connection header received
0928 14:45:44.662658 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.662661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.662706 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.662799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:44.663530 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.664077 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.664910 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.665138 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.665238 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.666727 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:44.666731 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:44.666738 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:44.666763 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:44.668172 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.668600 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.668608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.668609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.668683 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.668948 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.668951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.668953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.669191 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:44.669334 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.669565 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.669682 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 14:45:44.675444 14776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:45:44.678705 14776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:44.695082 15250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.689149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41399 (local address 127.0.0.1:55326)
0928 14:45:44.689636 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.689652 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.689738 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.690526 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.690528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.690569 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:44.690684 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.690695 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.691997 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.692000 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.693451 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.693461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.694652 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.694655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.694771 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.694794 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.694867 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.694933 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"mutex_wait_us":65,"thread_start_us":103,"threads_started":1}
I20260928 14:45:44.695096 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.688283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60004 (local address 127.13.223.67:38269)
0928 14:45:44.689009 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.689020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.689042 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:45:44.689283 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.689291 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.689587 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.689835 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:44.690310 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.691738 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.693350 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.694236 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.694355 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.694574 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.694794 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.694912 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":124,"threads_started":1}
I20260928 14:45:44.695124 15246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.688157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:38269 (local address 127.0.0.1:60004)
0928 14:45:44.688829 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.688888 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.689034 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.690023 (+ 989us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.690028 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.690048 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:44.690175 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.690181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.691866 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.691869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.693214 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.693225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.694383 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.694389 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.694532 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.694612 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.694750 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.694899 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":101,"threads_started":1}
I20260928 14:45:44.695529 15248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.688352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34651 (local address 127.0.0.1:47152)
0928 14:45:44.689184 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.689197 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.689296 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.691091 (+ 1795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.691094 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.691114 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:44.691219 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.691226 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.692840 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.692843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.693916 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.693924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.695086 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.695090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.695206 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.695231 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.695287 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.695354 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":112,"threads_started":1}
I20260928 14:45:44.695843 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.689613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55326 (local address 127.13.223.65:41399)
0928 14:45:44.690131 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.690138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.690158 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:45:44.690223 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.690227 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.690389 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.690580 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:44.690821 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.691860 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.693592 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.694373 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.694469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.694613 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.695595 (+ 982us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.695709 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":84,"threads_started":1}
I20260928 14:45:44.696995 15249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.688519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.13.223.66:34651)
0928 14:45:44.690521 (+ 2002us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.690535 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.690599 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:45:44.690733 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.690739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.690939 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.691104 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:44.691320 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.692680 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.694057 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.694940 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.696523 (+ 1583us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.696659 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.696734 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.696845 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1874,"thread_start_us":152,"threads_started":1}
I20260928 14:45:44.698644 15169 tablet_service.cc:1467] Processing CreateTablet for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4 (DEFAULT_TABLE table=test-table [id=d75bb7e5182b4f9a8d439a2b6c0401cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:44.699066 14901 tablet_service.cc:1467] Processing CreateTablet for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4 (DEFAULT_TABLE table=test-table [id=d75bb7e5182b4f9a8d439a2b6c0401cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:44.699908 15169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:44.699810 15035 tablet_service.cc:1467] Processing CreateTablet for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4 (DEFAULT_TABLE table=test-table [id=d75bb7e5182b4f9a8d439a2b6c0401cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:44.700227 14901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:44.700779 15035 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:44.707360 15252 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Bootstrap starting.
I20260928 14:45:44.708457 15253 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Bootstrap starting.
I20260928 14:45:44.708851 15254 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Bootstrap starting.
I20260928 14:45:44.709978 15252 tablet_bootstrap.cc:654] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:44.710880 15253 tablet_bootstrap.cc:654] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:44.710992 15252 log.cc:824] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:44.711290 15254 tablet_bootstrap.cc:654] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:44.712299 15253 log.cc:824] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:44.712299 15254 log.cc:824] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:44.713510 15252 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: No bootstrap required, opened a new log
I20260928 14:45:44.713744 15252 ts_tablet_manager.cc:1397] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:45:44.714310 15254 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: No bootstrap required, opened a new log
I20260928 14:45:44.714506 15254 ts_tablet_manager.cc:1397] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:45:44.715363 15253 tablet_bootstrap.cc:492] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: No bootstrap required, opened a new log
I20260928 14:45:44.715615 15253 ts_tablet_manager.cc:1397] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:45:44.718798 15254 raft_consensus.cc:348] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.719206 15254 raft_consensus.cc:374] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:44.719311 15254 raft_consensus.cc:729] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e3cc95f05da47218bca5da015af2852, State: Initialized, Role: FOLLOWER
I20260928 14:45:44.719643 15253 raft_consensus.cc:348] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.719916 15253 raft_consensus.cc:374] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:44.719861 15254 consensus_queue.cc:260] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.719997 15253 raft_consensus.cc:729] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f3209d20e8a4836a4da5602779d93e9, State: Initialized, Role: FOLLOWER
I20260928 14:45:44.720472 15253 consensus_queue.cc:260] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.720480 15252 raft_consensus.cc:348] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.720762 15101 heartbeater.cc:503] Master 127.13.223.126:42925 was elected leader, sending a full tablet report...
I20260928 14:45:44.720816 15252 raft_consensus.cc:374] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:44.720870 15252 raft_consensus.cc:729] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d7f060e8134464b5cfc1376ea5cace, State: Initialized, Role: FOLLOWER
I20260928 14:45:44.721077 15254 ts_tablet_manager.cc:1428] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 14:45:44.721311 15252 consensus_queue.cc:260] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.721498 15253 ts_tablet_manager.cc:1428] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:45:44.721858 15235 heartbeater.cc:503] Master 127.13.223.126:42925 was elected leader, sending a full tablet report...
I20260928 14:45:44.722038 14967 heartbeater.cc:503] Master 127.13.223.126:42925 was elected leader, sending a full tablet report...
I20260928 14:45:44.722383 15252 ts_tablet_manager.cc:1428] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
W20260928 14:45:44.877542 15236 tablet.cc:2369] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:44.880627 15102 tablet.cc:2369] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:44.887573 14968 tablet.cc:2369] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:45:44.943740 15258 raft_consensus.cc:482] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:44.944051 15258 raft_consensus.cc:504] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.945521 15258 leader_election.cc:288] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f3209d20e8a4836a4da5602779d93e9 (127.13.223.67:38269), f9d7f060e8134464b5cfc1376ea5cace (127.13.223.65:41399)
I20260928 14:45:44.950397 15247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.945969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:56843 (local address 127.13.223.67:38269)
0928 14:45:44.946108 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.946112 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.946309 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:45:44.946394 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.946399 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.946457 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.946584 (+ 127us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:44.946931 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.947559 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.948978 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.949714 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.949868 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.949945 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.950193 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.950274 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:45:44.950414 15261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.945850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:38269 (local address 127.13.223.66:56843)
0928 14:45:44.946145 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.946161 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.946293 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.946610 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.946614 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.946637 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:44.946791 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.946803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.947670 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.947673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.948804 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.948815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.949870 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.949873 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.950000 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.950029 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.950127 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.950231 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
I20260928 14:45:44.951323 15262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.946301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41399 (local address 127.13.223.66:35633)
0928 14:45:44.946731 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:44.946748 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:44.946849 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:44.947251 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:44.947254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:44.947271 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:44.947396 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.947402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.948476 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.948479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.949867 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:44.949880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.950952 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.950955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:44.951053 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.951075 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:45:44.951150 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:45:44.951207 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":73,"threads_started":1}
I20260928 14:45:44.951225 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ca6c3e970c4ed7a98b4bac8d57d9e4" candidate_uuid: "4e3cc95f05da47218bca5da015af2852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3209d20e8a4836a4da5602779d93e9" is_pre_election: true
I20260928 14:45:44.951393 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.946581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:35633 (local address 127.13.223.65:41399)
0928 14:45:44.946720 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.946724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.946833 (+ 109us) server_negotiation.cc:407] Connection header received
0928 14:45:44.947015 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.947020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.947076 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.947231 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:44.947516 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.948313 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.950020 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.950774 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.950928 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.950982 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.951187 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.951277 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:45:44.951697 15189 raft_consensus.cc:2454] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3cc95f05da47218bca5da015af2852 in term 0.
I20260928 14:45:44.952040 14921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ca6c3e970c4ed7a98b4bac8d57d9e4" candidate_uuid: "4e3cc95f05da47218bca5da015af2852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d7f060e8134464b5cfc1376ea5cace" is_pre_election: true
I20260928 14:45:44.952281 14989 leader_election.cc:302] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e3cc95f05da47218bca5da015af2852, 5f3209d20e8a4836a4da5602779d93e9; no voters:
I20260928 14:45:44.952348 14921 raft_consensus.cc:2454] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e3cc95f05da47218bca5da015af2852 in term 0.
I20260928 14:45:44.952697 15258 raft_consensus.cc:2781] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:44.952800 15258 raft_consensus.cc:482] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:44.952860 15258 raft_consensus.cc:3037] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:44.953981 15258 raft_consensus.cc:504] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.954499 15258 leader_election.cc:288] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [CANDIDATE]: Term 1 election: Requested vote from peers 5f3209d20e8a4836a4da5602779d93e9 (127.13.223.67:38269), f9d7f060e8134464b5cfc1376ea5cace (127.13.223.65:41399)
I20260928 14:45:44.955018 15189 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ca6c3e970c4ed7a98b4bac8d57d9e4" candidate_uuid: "4e3cc95f05da47218bca5da015af2852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f3209d20e8a4836a4da5602779d93e9"
I20260928 14:45:44.955194 15189 raft_consensus.cc:3037] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:44.955265 14921 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13ca6c3e970c4ed7a98b4bac8d57d9e4" candidate_uuid: "4e3cc95f05da47218bca5da015af2852" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f9d7f060e8134464b5cfc1376ea5cace"
I20260928 14:45:44.955397 14921 raft_consensus.cc:3037] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:44.956734 15189 raft_consensus.cc:2454] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 5f3209d20e8a4836a4da5602779d93e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3cc95f05da47218bca5da015af2852 in term 1.
I20260928 14:45:44.956734 14921 raft_consensus.cc:2454] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P f9d7f060e8134464b5cfc1376ea5cace [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e3cc95f05da47218bca5da015af2852 in term 1.
I20260928 14:45:44.957151 14989 leader_election.cc:302] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e3cc95f05da47218bca5da015af2852, 5f3209d20e8a4836a4da5602779d93e9; no voters:
I20260928 14:45:44.957361 15258 raft_consensus.cc:2781] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:44.957635 15258 raft_consensus.cc:686] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [term 1 LEADER]: Becoming Leader. State: Replica: 4e3cc95f05da47218bca5da015af2852, State: Running, Role: LEADER
I20260928 14:45:44.958072 15258 consensus_queue.cc:237] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } }
I20260928 14:45:44.962126 14776 catalog_manager.cc:5495] T 13ca6c3e970c4ed7a98b4bac8d57d9e4 P 4e3cc95f05da47218bca5da015af2852 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e3cc95f05da47218bca5da015af2852 (127.13.223.66). New cstate: current_term: 1 leader_uuid: "4e3cc95f05da47218bca5da015af2852" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f3209d20e8a4836a4da5602779d93e9" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 38269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9d7f060e8134464b5cfc1376ea5cace" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e3cc95f05da47218bca5da015af2852" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34651 } health_report { overall_health: HEALTHY } } }
Sep 28 14:45:44 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606743, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:44.994496 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:44.984131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55342 (local address 127.13.223.65:41399)
0928 14:45:44.984299 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:44.984304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:44.984490 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:45:44.984656 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:44.984659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:44.984702 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:44.984793 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:44.985536 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.986086 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.987063 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:44.987296 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:44.987406 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:44.990102 (+ 2696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:44.990118 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:44.990135 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:44.990165 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:44.992153 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.992660 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.992667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.992669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.992785 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:44.993303 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:44.993307 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:44.993308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:44.993613 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:44.993796 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:44.994129 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:45:44.994288 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606745, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:45.025655 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:45.015600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.13.223.65:41399)
0928 14:45:45.015798 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:45.015804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:45.015959 (+ 155us) server_negotiation.cc:407] Connection header received
0928 14:45:45.016127 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:45.016136 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:45.016180 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:45.016271 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:45.017068 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:45.017597 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:45.018453 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:45.018787 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:45.018904 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:45.021778 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:45.021782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:45.021785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:45.021819 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:45.023509 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:45.024025 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:45.024029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:45.024031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:45.024136 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:45.024455 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:45.024459 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:45.024461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:45.024672 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:45.024833 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:45.025254 (+ 421us) server_negotiation.cc:299] Negotiation successful
0928 14:45:45.025399 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[14731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[14731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606745, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:45.055008 15251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:45.045182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.13.223.65:41399)
0928 14:45:45.045354 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:45.045358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:45.045427 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:45:45.045602 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:45.045605 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:45.045651 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:45.045725 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:45.046756 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:45.047464 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:45.048387 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:45.048618 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:45.048711 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:45.051476 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:45.051480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:45.051482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:45.051512 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:45.052893 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:45.053397 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:45.053401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:45.053403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:45.053503 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:45.053787 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:45.053790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:45.053791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:45.053985 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:45.054112 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:45.054396 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:45:45.054803 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260928 14:45:45.056473 14881 tablet_service.cc:2908] Rejecting scan request for tablet 13ca6c3e970c4ed7a98b4bac8d57d9e4: Uninitialized: clean time has not yet been initialized
I20260928 14:45:45.059309 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14838
I20260928 14:45:45.067384 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14972
I20260928 14:45:45.073916 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15106
I20260928 14:45:45.079943 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 14744
2026-09-28T14:45:45Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2673 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[15277](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[15277](info): set up 2 sockets
Sep 28 14:45:45 dist-test-slave-gkhx krb5kdc[15277](info): commencing operation
krb5kdc: starting...
W20260928 14:45:46.181131 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:46 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:46.323449 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46691
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:46.441409 15290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:46.441773 15290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:46.441818 15290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:46.446287 15290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:46.446378 15290 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:45:46.446398 15290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:46.446416 15290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:46.446434 15290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:46.446449 15290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:46.452358 15290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46691
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:46.453614 15290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:46.454821 15290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:46.465015 15296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:46.465001 15298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:46.465085 15290 server_base.cc:1034] running on GCE node
W20260928 14:45:46.465003 15295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:46.466005 15290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:46.467535 15290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:46.468720 15290 hybrid_clock.cc:648] HybridClock initialized: now 1790606746468683 us; error 58 us; skew 500 ppm
Sep 28 14:45:46 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:46.471907 15290 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:46.473075 15290 webserver.cc:460] Webserver started at http://127.13.223.126:34369/ using document root <none> and password file <none>
I20260928 14:45:46.473704 15290 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:46.473804 15290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:46.474056 15290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:46.476938 15290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "9253c621b33a41cd8abcbedda9a31c5c"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "a31dca1a798dac5e05a3cbaddbcaf0c9"
server_key_iv: "2ceebc8d9f89023d10b7f2b85fc0b093"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.477633 15290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "9253c621b33a41cd8abcbedda9a31c5c"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "a31dca1a798dac5e05a3cbaddbcaf0c9"
server_key_iv: "2ceebc8d9f89023d10b7f2b85fc0b093"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.481901 15290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 14:45:46.484786 15305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:46.486253 15290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:46.486431 15290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "9253c621b33a41cd8abcbedda9a31c5c"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "a31dca1a798dac5e05a3cbaddbcaf0c9"
server_key_iv: "2ceebc8d9f89023d10b7f2b85fc0b093"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.486591 15290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:46.502501 15290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:46.506140 15290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:46.506364 15290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:46.514994 15290 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:46691
I20260928 14:45:46.515009 15367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:46691 every 8 connection(s)
I20260928 14:45:46.516014 15290 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:46.519104 15368 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:46.521194 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15290
I20260928 14:45:46.521291 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:46.521561 14205 external_mini_cluster.cc:1428] Setting key 8937e03053a786742f89e187f1e0dae3
I20260928 14:45:46.526574 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Bootstrap starting.
I20260928 14:45:46.529395 15368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:46.530323 15368 log.cc:824] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:45:46 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:46.532840 15368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: No bootstrap required, opened a new log
I20260928 14:45:46.537348 15368 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } }
I20260928 14:45:46.537729 15368 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:46.537802 15368 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9253c621b33a41cd8abcbedda9a31c5c, State: Initialized, Role: FOLLOWER
I20260928 14:45:46.538303 15368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } }
I20260928 14:45:46.538466 15368 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:46.538533 15368 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:46.538657 15368 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:46.538909 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:46.523162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34950 (local address 127.13.223.126:46691)
0928 14:45:46.523616 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:46.523636 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:46.523682 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:45:46.524547 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:46.524576 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:46.525069 (+ 493us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:46.525476 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:46.526207 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.527720 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.528812 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.529621 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.529813 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:46.532682 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:46.532689 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:46.532707 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:46.532737 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:46.535570 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.536047 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.536053 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.536060 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.536211 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.536497 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.536502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.536505 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.537074 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:46.537278 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:46.537487 (+ 209us) server_negotiation.cc:299] Negotiation successful
0928 14:45:46.537819 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":118,"threads_started":1}
I20260928 14:45:46.539743 15368 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } }
I20260928 14:45:46.540164 15368 leader_election.cc:302] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9253c621b33a41cd8abcbedda9a31c5c; no voters:
I20260928 14:45:46.540546 15368 leader_election.cc:288] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:46.540707 15373 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:46.540998 15373 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [term 1 LEADER]: Becoming Leader. State: Replica: 9253c621b33a41cd8abcbedda9a31c5c, State: Running, Role: LEADER
I20260928 14:45:46.541391 15373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } }
I20260928 14:45:46.541963 15368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:46.542644 15373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9253c621b33a41cd8abcbedda9a31c5c. Latest consensus state: current_term: 1 leader_uuid: "9253c621b33a41cd8abcbedda9a31c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } } }
I20260928 14:45:46.542853 15373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:46.543131 15374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9253c621b33a41cd8abcbedda9a31c5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9253c621b33a41cd8abcbedda9a31c5c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46691 } } }
I20260928 14:45:46.543286 15381 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:46.543305 15374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:46.546811 15381 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:46.553431 15381 catalog_manager.cc:1347] Generated new cluster ID: 8b265b760a44426fac2c9927c04b47c9
I20260928 14:45:46.553534 15381 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:46.570104 15381 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:46.571305 15381 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:46.582491 15381 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Generated new TSK 0
I20260928 14:45:46.583547 15381 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:45:46.623124 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46691
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:46.757826 15394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:46.758138 15394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:46.758188 15394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:46.762593 15394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:46.762676 15394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:46.762717 15394 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:46.762789 15394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:45:46.768575 15394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:46.769820 15394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:46.771025 15394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:46.782773 15402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:46.782794 15399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:46.782930 15400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:46.783035 15394 server_base.cc:1034] running on GCE node
I20260928 14:45:46.783640 15394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:46.784400 15394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:46.785606 15394 hybrid_clock.cc:648] HybridClock initialized: now 1790606746785591 us; error 45 us; skew 500 ppm
Sep 28 14:45:46 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:46.789137 15394 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:45:46.790336 15394 webserver.cc:460] Webserver started at http://127.13.223.65:40279/ using document root <none> and password file <none>
I20260928 14:45:46.791127 15394 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:46.791239 15394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:46.791535 15394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:46.794078 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "789d4b5499b0707ac1336e1bf51713cc"
server_key_iv: "bdbbaec70a480d4648adac450243c120"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.794845 15394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "789d4b5499b0707ac1336e1bf51713cc"
server_key_iv: "bdbbaec70a480d4648adac450243c120"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.799418 15394 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:45:46.802802 15409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:46.804354 15394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:46.804592 15394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
format_stamp: "Formatted at 2026-09-28 14:45:46 on dist-test-slave-gkhx"
server_key: "789d4b5499b0707ac1336e1bf51713cc"
server_key_iv: "bdbbaec70a480d4648adac450243c120"
server_key_version: "encryptionkey@0"
I20260928 14:45:46.804751 15394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:46.833000 15394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:46.836730 15394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:46.836982 15394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:46.838708 15394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:46.838789 15394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:46.838877 15394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:46.838924 15394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:45:46 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:46.851220 15394 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:42157
I20260928 14:45:46.851298 15524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:42157 every 8 connection(s)
I20260928 14:45:46.852662 15394 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:45:46.855650 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:46.839730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:42423 (local address 127.13.223.126:46691)
0928 14:45:46.839902 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:46.839908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:46.840915 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 14:45:46.842197 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:46.842201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:46.842266 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:46.842367 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:46.844511 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.845180 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.847382 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.847683 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.847778 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:46.851197 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:46.851202 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:46.851205 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:46.851245 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:46.853241 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.853981 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.853984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.853986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.854090 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.854432 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.854434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.854435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.854720 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:46.854804 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:46.855398 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 14:45:46.855522 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:45:46.856671 15418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:46.840220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.65:42423)
0928 14:45:46.840721 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:46.840776 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:46.841848 (+ 1072us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:46.842714 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:46.842728 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:46.843358 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:46.844289 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:46.844320 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.845310 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.845314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:46.847220 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:46.847236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.847865 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.847870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:46.848157 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:46.848957 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:46.848978 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:46.850906 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:46.853408 (+ 2502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:46.853418 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:46.853442 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:46.853842 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:46.854185 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:46.854188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:46.854190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:46.854338 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:46.854840 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:46.854846 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:46.855132 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 14:45:46.855315 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 14:45:46.855632 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":145,"threads_started":1}
I20260928 14:45:46.858362 15525 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46691
I20260928 14:45:46.858610 15525 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:46.859316 15525 heartbeater.cc:511] Master 127.13.223.126:46691 requested a full tablet report, sending...
I20260928 14:45:46.861099 15320 ts_manager.cc:194] Registered new tserver with Master: 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157)
I20260928 14:45:46.862056 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15394
I20260928 14:45:46.862181 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:45:46.862442 14205 external_mini_cluster.cc:1428] Setting key 52b7617eb39a5a50eb194431df3d39e6
I20260928 14:45:46.862648 15320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:42423
I20260928 14:45:46.869251 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:46.860737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:35901 (local address 127.13.223.126:46691)
0928 14:45:46.860859 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:46.860863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:46.861160 (+ 297us) server_negotiation.cc:407] Connection header received
0928 14:45:46.861295 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:46.861298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:46.861349 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:46.861436 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:46.861995 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.862788 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.863925 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.864235 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.864345 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:46.865921 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:46.865933 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:46.865935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:46.865959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:46.867458 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.867979 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.867983 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.867985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.868072 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:46.868368 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:46.868371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:46.868373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:46.868572 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:46.868681 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:46.868947 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:45:46.869066 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:45:46.869266 15531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:46.860646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.65:35901)
0928 14:45:46.861073 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:46.861086 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:46.861206 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:46.861455 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:46.861457 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:46.861687 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:46.861852 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:46.861862 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.862956 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.862960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:46.863782 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:46.863810 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:46.864378 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:46.864385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:46.864603 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:46.865224 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:46.865245 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:46.865778 (+ 533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:46.867645 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:46.867649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:46.867651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:46.867871 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:46.868167 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:46.868171 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:46.868173 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:46.868266 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:46.868670 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:46.868673 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:46.868782 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:45:46.868890 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 14:45:46.869070 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":22,"thread_start_us":250,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:45:46.895021 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46691
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:47.017911 15534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:47.018234 15534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:47.018280 15534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:47.022657 15534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:47.022737 15534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:47.022778 15534 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:47.022814 15534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:45:47.029052 15534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:47.030303 15534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:47.031704 15534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:47.043401 15534 server_base.cc:1034] running on GCE node
W20260928 14:45:47.043318 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:47.043537 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:47.043306 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:47.044183 15534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:47.044952 15534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:47.046226 15534 hybrid_clock.cc:648] HybridClock initialized: now 1790606747046191 us; error 52 us; skew 500 ppm
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:47.049672 15534 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:45:47.050985 15534 webserver.cc:460] Webserver started at http://127.13.223.66:41893/ using document root <none> and password file <none>
I20260928 14:45:47.051594 15534 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:47.051663 15534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:47.051921 15534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:47.054536 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "2a279907577a4281ae757d9f4015e0ce"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "4b6701c36fbe5612633af1bc372890e5"
server_key_iv: "97b9001998271740048f74beb38c935a"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.055390 15534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "2a279907577a4281ae757d9f4015e0ce"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "4b6701c36fbe5612633af1bc372890e5"
server_key_iv: "97b9001998271740048f74beb38c935a"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.060061 15534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:45:47.063521 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:47.065201 15534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:47.065376 15534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "2a279907577a4281ae757d9f4015e0ce"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "4b6701c36fbe5612633af1bc372890e5"
server_key_iv: "97b9001998271740048f74beb38c935a"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.065526 15534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:47.084445 15534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:47.088589 15534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:47.088819 15534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:47.090720 15534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:47.090787 15534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:47.090854 15534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:47.090886 15534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:47.102785 15534 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37419
I20260928 14:45:47.102843 15664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37419 every 8 connection(s)
I20260928 14:45:47.104182 15534 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:45:47.108273 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.091835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:60535 (local address 127.13.223.126:46691)
0928 14:45:47.092010 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.092017 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.093109 (+ 1092us) server_negotiation.cc:407] Connection header received
0928 14:45:47.094604 (+ 1495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.094611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.094697 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.094817 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.096424 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.097044 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.098483 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.098825 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.098970 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.102811 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.102816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.102819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.102863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.105086 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.105770 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.105774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.105776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.105879 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.106642 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.106645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.106646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.107150 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.107286 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.107993 (+ 707us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.108140 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:45:47.109241 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.092333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.66:60535)
0928 14:45:47.092876 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.092949 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.094195 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.095058 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.095067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.095622 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:47.096245 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.096261 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.097192 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.097196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.098326 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.098338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.099035 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.099040 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.099237 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.099959 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:47.099979 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:47.102465 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:47.105285 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.105292 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.105311 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.105631 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.106185 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.106188 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.106189 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.106419 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.107342 (+ 923us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:47.107348 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:47.107673 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.107863 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.108187 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":160,"threads_started":1}
I20260928 14:45:47.111094 15665 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46691
I20260928 14:45:47.111332 15665 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:47.112147 15665 heartbeater.cc:511] Master 127.13.223.126:46691 requested a full tablet report, sending...
I20260928 14:45:47.113277 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15534
I20260928 14:45:47.113394 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:45:47.113597 15320 ts_manager.cc:194] Registered new tserver with Master: 2a279907577a4281ae757d9f4015e0ce (127.13.223.66:37419)
I20260928 14:45:47.113777 14205 external_mini_cluster.cc:1428] Setting key 614d2be945947c384910db961d02bacf
I20260928 14:45:47.114367 15320 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:60535
I20260928 14:45:47.122377 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.113423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:49785 (local address 127.13.223.126:46691)
0928 14:45:47.113563 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.113570 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.113907 (+ 337us) server_negotiation.cc:407] Connection header received
0928 14:45:47.114036 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.114039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.114089 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.114173 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.114943 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.115631 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.116529 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.116780 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.116897 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.118151 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.118155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.118159 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.118184 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.120130 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.120768 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.120771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.120772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.120870 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.121275 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.121279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.121287 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.121568 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.121681 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.122096 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.122241 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:45:47.122429 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.113293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.66:49785)
0928 14:45:47.113786 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.113804 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.113933 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.114328 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.114331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.114657 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:47.114801 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.114810 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.115785 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.115791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.116399 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.116408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.116903 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.116907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.117028 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.117621 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:47.117634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:47.117992 (+ 358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:47.120283 (+ 2291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.120288 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.120290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.120628 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.120997 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.121004 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.121007 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.121130 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.121691 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:47.121696 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:47.121844 (+ 148us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.121975 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.122176 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:45:47.147320 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46691
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:47.280344 15674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:47.280687 15674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:47.280737 15674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:47.285300 15674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:47.285390 15674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:47.285434 15674 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:47.285470 15674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:45:47.291612 15674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:46691
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:47.292882 15674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:47.294076 15674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:47.305029 15679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:47.305204 15674 server_base.cc:1034] running on GCE node
W20260928 14:45:47.305058 15680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:47.305447 15682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:47.305876 15674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:47.306660 15674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:47.307897 15674 hybrid_clock.cc:648] HybridClock initialized: now 1790606747307837 us; error 63 us; skew 500 ppm
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:47.311487 15674 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:45:47.312568 15674 webserver.cc:460] Webserver started at http://127.13.223.67:40641/ using document root <none> and password file <none>
I20260928 14:45:47.313232 15674 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:47.313337 15674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:47.313598 15674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:47.316565 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "226cec9290914bcd8a249c029b40030e"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "41976067b5c3aced218e1089164c95cb"
server_key_iv: "401da6a809a5ce2934b64ec9839533cd"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.317381 15674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "226cec9290914bcd8a249c029b40030e"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "41976067b5c3aced218e1089164c95cb"
server_key_iv: "401da6a809a5ce2934b64ec9839533cd"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.322885 15674 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260928 14:45:47.327066 15689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:47.328892 15674 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:45:47.329136 15674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "226cec9290914bcd8a249c029b40030e"
format_stamp: "Formatted at 2026-09-28 14:45:47 on dist-test-slave-gkhx"
server_key: "41976067b5c3aced218e1089164c95cb"
server_key_iv: "401da6a809a5ce2934b64ec9839533cd"
server_key_version: "encryptionkey@0"
I20260928 14:45:47.329301 15674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:47.346925 15674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:47.351141 15674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:47.351418 15674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:47.353201 15674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:47.353298 15674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:47.353382 15674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:47.353432 15674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:47.365971 15674 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:33905
I20260928 14:45:47.365993 15804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:33905 every 8 connection(s)
I20260928 14:45:47.367231 15674 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:45:47.370710 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.354487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:54895 (local address 127.13.223.126:46691)
0928 14:45:47.354669 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.354673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.356408 (+ 1735us) server_negotiation.cc:407] Connection header received
0928 14:45:47.357835 (+ 1427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.357839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.357907 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.358008 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.360258 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.361195 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.362684 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.362939 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.363066 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.366240 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.366244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.366246 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.366288 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.368143 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.368792 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.368795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.368797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.368886 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.369260 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.369263 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.369265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.369481 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.369573 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.370341 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.370481 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:45:47.371603 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.355289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.67:54895)
0928 14:45:47.355971 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.356263 (+ 292us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.357516 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.358280 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.358294 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.359082 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:47.360013 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.360041 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.361369 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.361373 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.362497 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.362509 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.363128 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.363132 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.363344 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.364184 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:47.364207 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:47.365975 (+ 1768us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:47.368276 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.368282 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.368299 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.368661 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.368980 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.368983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.368984 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.369134 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.369630 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:47.369691 (+ 61us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:47.370025 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.370215 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.370531 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":162,"threads_started":1}
I20260928 14:45:47.373348 15805 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46691
I20260928 14:45:47.373605 15805 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:47.374193 15805 heartbeater.cc:511] Master 127.13.223.126:46691 requested a full tablet report, sending...
I20260928 14:45:47.375523 15322 ts_manager.cc:194] Registered new tserver with Master: 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905)
I20260928 14:45:47.375800 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15674
I20260928 14:45:47.375924 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:45:47.376236 14205 external_mini_cluster.cc:1428] Setting key 6bbd4a4d9fe986c70ba43aa33c66bfe1
I20260928 14:45:47.376533 15322 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:54895
I20260928 14:45:47.378774 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:45:47.388402 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.376602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:43309 (local address 127.13.223.126:46691)
0928 14:45:47.376719 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.376724 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.376741 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:45:47.376993 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.376996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.377044 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.377128 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.377723 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.378586 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.379902 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.380373 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.380514 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.382020 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.382027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.382035 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.382074 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.384341 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.386338 (+ 1997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.386343 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.386344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.386473 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.387117 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.387124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.387126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.387444 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.387587 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.388067 (+ 480us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.388204 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:45:47.388417 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.376144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.13.223.67:43309)
0928 14:45:47.376510 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.376547 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.376722 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.377159 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.377162 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.377428 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:47.377609 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.377620 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.378733 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.378737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.379750 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.379762 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.380529 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.380532 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.380639 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.381416 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:47.381430 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:47.381832 (+ 402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:47.384546 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.384549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.384550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.384789 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.386730 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.386737 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.386741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.386900 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.387660 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:47.387666 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:47.387810 (+ 144us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.387956 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.388173 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":124,"threads_started":1}
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:47.412175 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.401663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.13.223.126:46691)
0928 14:45:47.401813 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.401817 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.401832 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:45:47.401876 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.401879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.401924 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.402094 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.403052 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.403815 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.404829 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.405058 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.405212 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.407924 (+ 2712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.407928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.407931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.407963 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.409754 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.410460 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.410463 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.410465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.410617 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.410960 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.410963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.410964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.411260 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.411382 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.411760 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.411880 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:45:47.416162 15322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:45:47.419420 15322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:47.436204 15822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.428339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37419 (local address 127.0.0.1:35084)
0928 14:45:47.429063 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.429139 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.429861 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.430636 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.430639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.430653 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.430728 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.430736 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432332 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.432335 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.434155 (+ 1820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.434169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.435473 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.435478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.435641 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.435705 (+ 64us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.435818 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.436024 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":165,"thread_start_us":87,"threads_started":1}
I20260928 14:45:47.436347 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.429208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49282 (local address 127.13.223.65:42157)
0928 14:45:47.430066 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.430072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.430095 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:45:47.430184 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.430190 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.430533 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.430758 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.431013 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432227 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.434023 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.435261 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.435394 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.435586 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.435914 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.436158 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":90,"threads_started":1}
I20260928 14:45:47.436896 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.428535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:33905 (local address 127.0.0.1:39426)
0928 14:45:47.429295 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.429308 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.429428 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.430466 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.430475 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.430503 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.430697 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.430706 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432320 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.432324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.434157 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.434169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.436462 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.436466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.436568 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.436590 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.436662 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.436744 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":680,"mutex_wait_us":199,"thread_start_us":141,"threads_started":1}
I20260928 14:45:47.437069 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.428726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42157 (local address 127.0.0.1:49282)
0928 14:45:47.429726 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.429739 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.429821 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.430737 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.430741 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.430762 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.430860 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.430870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432381 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.432385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.433873 (+ 1488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.433886 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.435651 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.435655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.435749 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.435768 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.435839 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.436885 (+ 1046us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":930,"mutex_wait_us":932,"thread_start_us":95,"threads_started":1}
I20260928 14:45:47.437175 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.428537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35084 (local address 127.13.223.66:37419)
0928 14:45:47.430087 (+ 1550us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.430096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.430116 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:45:47.430190 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.430194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.430361 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.430540 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.430973 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432155 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.434360 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.435261 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.436623 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.436829 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.436891 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.437012 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1446,"thread_start_us":108,"threads_started":1}
I20260928 14:45:47.437393 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.429405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.13.223.67:33905)
0928 14:45:47.429749 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.429760 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.429781 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:45:47.429889 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.429898 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.430158 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.430467 (+ 309us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.430850 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.432157 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.434338 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.436080 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.436204 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.436383 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.437134 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.437246 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":134,"threads_started":1}
I20260928 14:45:47.440627 15459 tablet_service.cc:1467] Processing CreateTablet for tablet 29844138ddff4e3ba700aca75c28f627 (DEFAULT_TABLE table=test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.441165 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 29844138ddff4e3ba700aca75c28f627 (DEFAULT_TABLE table=test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.441150 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 29844138ddff4e3ba700aca75c28f627 (DEFAULT_TABLE table=test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.441974 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29844138ddff4e3ba700aca75c28f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.442199 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29844138ddff4e3ba700aca75c28f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.442210 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29844138ddff4e3ba700aca75c28f627. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.450070 15828 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Bootstrap starting.
I20260928 14:45:47.451016 15829 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Bootstrap starting.
I20260928 14:45:47.453478 15829 tablet_bootstrap.cc:654] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.453562 15828 tablet_bootstrap.cc:654] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.454306 15829 log.cc:824] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:47.454324 15828 log.cc:824] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:47.454974 15830 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Bootstrap starting.
I20260928 14:45:47.456823 15828 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: No bootstrap required, opened a new log
I20260928 14:45:47.457057 15828 ts_tablet_manager.cc:1397] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:45:47.457552 15830 tablet_bootstrap.cc:654] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.458649 15830 log.cc:824] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:47.458756 15829 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: No bootstrap required, opened a new log
I20260928 14:45:47.459146 15829 ts_tablet_manager.cc:1397] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 14:45:47.461560 15828 raft_consensus.cc:348] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.461848 15828 raft_consensus.cc:374] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.461895 15828 raft_consensus.cc:729] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226cec9290914bcd8a249c029b40030e, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.462301 15830 tablet_bootstrap.cc:492] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: No bootstrap required, opened a new log
I20260928 14:45:47.462417 15828 consensus_queue.cc:260] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.462652 15830 ts_tablet_manager.cc:1397] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 14:45:47.463219 15805 heartbeater.cc:503] Master 127.13.223.126:46691 was elected leader, sending a full tablet report...
I20260928 14:45:47.463650 15828 ts_tablet_manager.cc:1428] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:45:47.464843 15829 raft_consensus.cc:348] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.465198 15829 raft_consensus.cc:374] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.465260 15829 raft_consensus.cc:729] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d052656cf4d4ead5e8dc5fec230be, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.465770 15829 consensus_queue.cc:260] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.466632 15525 heartbeater.cc:503] Master 127.13.223.126:46691 was elected leader, sending a full tablet report...
I20260928 14:45:47.466953 15829 ts_tablet_manager.cc:1428] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 14:45:47.467080 15830 raft_consensus.cc:348] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.467419 15830 raft_consensus.cc:374] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.467489 15830 raft_consensus.cc:729] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a279907577a4281ae757d9f4015e0ce, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.467972 15830 consensus_queue.cc:260] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.469241 15665 heartbeater.cc:503] Master 127.13.223.126:46691 was elected leader, sending a full tablet report...
I20260928 14:45:47.469265 15830 ts_tablet_manager.cc:1428] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260928 14:45:47.604456 15526 tablet.cc:2369] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:47.605747 15666 tablet.cc:2369] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:47.618667 15806 tablet.cc:2369] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:45:47.727596 15834 raft_consensus.cc:482] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:47.727805 15834 raft_consensus.cc:504] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.729328 15834 leader_election.cc:288] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a279907577a4281ae757d9f4015e0ce (127.13.223.66:37419), 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157)
I20260928 14:45:47.734776 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.729576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37419 (local address 127.13.223.67:45825)
0928 14:45:47.729730 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.729748 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.729852 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.730644 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.730647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.730734 (+ 87us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.730905 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.730914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.732117 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.732120 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.733374 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.733385 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.734208 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.734212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.734367 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.734387 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.734483 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.734604 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":26752}
I20260928 14:45:47.734776 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.729863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:45825 (local address 127.13.223.66:37419)
0928 14:45:47.730188 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.730196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.730240 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:45:47.730327 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.730330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.730436 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.730558 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.731120 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.731974 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.733536 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.734074 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.734214 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.734280 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.734499 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.734633 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260928 14:45:47.735587 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.730056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42157 (local address 127.13.223.67:44349)
0928 14:45:47.730595 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.730611 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.730744 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.731175 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.731177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.731195 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.731927 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.731953 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.733148 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.733152 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.734090 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.734101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.735122 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.735126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.735245 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.735282 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.735377 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.735438 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"spinlock_wait_cycles":5120,"thread_start_us":139,"threads_started":1}
I20260928 14:45:47.735664 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.730175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44349 (local address 127.13.223.65:42157)
0928 14:45:47.730832 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.730838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.730885 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:45:47.730945 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.730977 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.731041 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.731121 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.732083 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.732993 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.734236 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.734969 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.735120 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.735188 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.735423 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.735525 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:45:47.735656 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29844138ddff4e3ba700aca75c28f627" candidate_uuid: "226cec9290914bcd8a249c029b40030e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a279907577a4281ae757d9f4015e0ce" is_pre_election: true
I20260928 14:45:47.736140 15619 raft_consensus.cc:2454] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226cec9290914bcd8a249c029b40030e in term 0.
I20260928 14:45:47.736658 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29844138ddff4e3ba700aca75c28f627" candidate_uuid: "226cec9290914bcd8a249c029b40030e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" is_pre_election: true
I20260928 14:45:47.736869 15693 leader_election.cc:302] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226cec9290914bcd8a249c029b40030e, 2a279907577a4281ae757d9f4015e0ce; no voters:
I20260928 14:45:47.737025 15479 raft_consensus.cc:2454] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 226cec9290914bcd8a249c029b40030e in term 0.
I20260928 14:45:47.737295 15834 raft_consensus.cc:2781] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:47.737394 15834 raft_consensus.cc:482] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:47.737460 15834 raft_consensus.cc:3037] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:47.738703 15834 raft_consensus.cc:504] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.739135 15834 leader_election.cc:288] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [CANDIDATE]: Term 1 election: Requested vote from peers 2a279907577a4281ae757d9f4015e0ce (127.13.223.66:37419), 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157)
I20260928 14:45:47.739655 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29844138ddff4e3ba700aca75c28f627" candidate_uuid: "226cec9290914bcd8a249c029b40030e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
I20260928 14:45:47.739676 15619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29844138ddff4e3ba700aca75c28f627" candidate_uuid: "226cec9290914bcd8a249c029b40030e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a279907577a4281ae757d9f4015e0ce"
I20260928 14:45:47.739780 15479 raft_consensus.cc:3037] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:47.739831 15619 raft_consensus.cc:3037] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:47.740940 15619 raft_consensus.cc:2454] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226cec9290914bcd8a249c029b40030e in term 1.
I20260928 14:45:47.740954 15479 raft_consensus.cc:2454] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 226cec9290914bcd8a249c029b40030e in term 1.
I20260928 14:45:47.741338 15693 leader_election.cc:302] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 226cec9290914bcd8a249c029b40030e, 2a279907577a4281ae757d9f4015e0ce; no voters:
I20260928 14:45:47.741557 15834 raft_consensus.cc:2781] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:47.741814 15834 raft_consensus.cc:686] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 1 LEADER]: Becoming Leader. State: Replica: 226cec9290914bcd8a249c029b40030e, State: Running, Role: LEADER
I20260928 14:45:47.742244 15834 consensus_queue.cc:237] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } }
I20260928 14:45:47.745905 15322 catalog_manager.cc:5495] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e reported cstate change: term changed from 0 to 1, leader changed from <none> to 226cec9290914bcd8a249c029b40030e (127.13.223.67). New cstate: current_term: 1 leader_uuid: "226cec9290914bcd8a249c029b40030e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:47.810740 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.806297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.13.223.67:33905)
0928 14:45:47.806443 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.806447 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.806468 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:45:47.806733 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.806736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.806794 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.806882 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:47.807259 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.807821 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.808779 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.809069 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.809154 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.809542 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:47.810164 (+ 622us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:47.810267 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.810421 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.810499 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:45:47.817546 15479 raft_consensus.cc:1261] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Refusing update from remote peer 226cec9290914bcd8a249c029b40030e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:47.817560 15619 raft_consensus.cc:1261] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Refusing update from remote peer 226cec9290914bcd8a249c029b40030e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:47.818470 15838 consensus_queue.cc:1035] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:47.818763 15834 consensus_queue.cc:1035] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:47.828047 15842 mvcc.cc:205] Tried to move back new op lower bound from 7334325239051345920 to 7334325238755618816. Current Snapshot: MvccSnapshot[applied={T|T < 7334325239051345920}]
I20260928 14:45:47.830459 15843 mvcc.cc:205] Tried to move back new op lower bound from 7334325239051345920 to 7334325238755618816. Current Snapshot: MvccSnapshot[applied={T|T < 7334325239051345920}]
I20260928 14:45:47.841301 15322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:47.841531 15322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:47.844633 15322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Sending DeleteTablet for 3 replicas of tablet 29844138ddff4e3ba700aca75c28f627
I20260928 14:45:47.845741 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 29844138ddff4e3ba700aca75c28f627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:47 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:35084
I20260928 14:45:47.845693 15739 tablet_service.cc:1514] Processing DeleteTablet for tablet 29844138ddff4e3ba700aca75c28f627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:47 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39426
I20260928 14:45:47.846172 15459 tablet_service.cc:1514] Processing DeleteTablet for tablet 29844138ddff4e3ba700aca75c28f627 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:47 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49282
I20260928 14:45:47.846262 15851 tablet_replica.cc:330] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: stopping tablet replica
I20260928 14:45:47.846626 15851 raft_consensus.cc:2229] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:47.846940 15853 tablet_replica.cc:330] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: stopping tablet replica
I20260928 14:45:47.847054 15851 raft_consensus.cc:2258] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:47.847306 15853 raft_consensus.cc:2229] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:47.847617 15853 raft_consensus.cc:2258] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:47.848183 15322 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:45:47.848606 15852 tablet_replica.cc:330] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: stopping tablet replica
W20260928 14:45:47.848683 15322 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:47.848908 15851 ts_tablet_manager.cc:1905] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:47.849143 15852 raft_consensus.cc:2229] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:45:47.849645 15852 raft_consensus.cc:2258] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:47.852125 15851 ts_tablet_manager.cc:1918] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:45:47.852252 15851 log.cc:1196] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/29844138ddff4e3ba700aca75c28f627
I20260928 14:45:47.852599 15851 ts_tablet_manager.cc:1939] T 29844138ddff4e3ba700aca75c28f627 P 2a279907577a4281ae757d9f4015e0ce: Deleting consensus metadata
I20260928 14:45:47.853473 15853 ts_tablet_manager.cc:1905] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:47.853685 15852 ts_tablet_manager.cc:1905] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:47.853973 15309 catalog_manager.cc:4841] TS 2a279907577a4281ae757d9f4015e0ce (127.13.223.66:37419): tablet 29844138ddff4e3ba700aca75c28f627 (table test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]) successfully deleted
I20260928 14:45:47.856173 15459 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 (DEFAULT_TABLE table=test-table [id=e96d6a4884304bdd837f4d047826d1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.856381 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 (DEFAULT_TABLE table=test-table [id=e96d6a4884304bdd837f4d047826d1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.856676 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad0b58063494950a8d3fdb35b0e2c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.856846 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad0b58063494950a8d3fdb35b0e2c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.859344 15829 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Bootstrap starting.
I20260928 14:45:47.859413 15828 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Bootstrap starting.
I20260928 14:45:47.860122 15853 ts_tablet_manager.cc:1918] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:45:47.860244 15853 log.cc:1196] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/29844138ddff4e3ba700aca75c28f627
I20260928 14:45:47.860373 15828 tablet_bootstrap.cc:654] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.860530 15829 tablet_bootstrap.cc:654] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.860695 15853 ts_tablet_manager.cc:1939] T 29844138ddff4e3ba700aca75c28f627 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting consensus metadata
I20260928 14:45:47.861992 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 (DEFAULT_TABLE table=test-table [id=e96d6a4884304bdd837f4d047826d1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:47.862334 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ad0b58063494950a8d3fdb35b0e2c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.862486 15829 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: No bootstrap required, opened a new log
I20260928 14:45:47.862619 15829 ts_tablet_manager.cc:1397] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:47.862620 15852 ts_tablet_manager.cc:1918] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:45:47.862727 15852 log.cc:1196] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/29844138ddff4e3ba700aca75c28f627
I20260928 14:45:47.863047 15852 ts_tablet_manager.cc:1939] T 29844138ddff4e3ba700aca75c28f627 P 226cec9290914bcd8a249c029b40030e: Deleting consensus metadata
I20260928 14:45:47.863133 15829 raft_consensus.cc:348] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.863255 15829 raft_consensus.cc:374] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.863286 15829 raft_consensus.cc:729] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d052656cf4d4ead5e8dc5fec230be, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.863399 15829 consensus_queue.cc:260] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.863686 15829 ts_tablet_manager.cc:1428] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:45:47.864293 15306 catalog_manager.cc:4841] TS 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157): tablet 29844138ddff4e3ba700aca75c28f627 (table test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]) successfully deleted
I20260928 14:45:47.865123 15830 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Bootstrap starting.
I20260928 14:45:47.866293 15830 tablet_bootstrap.cc:654] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.866605 15308 catalog_manager.cc:4841] TS 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905): tablet 29844138ddff4e3ba700aca75c28f627 (table test-table [id=2cfc7b116f8e44ef9e319f5fd86562c3]) successfully deleted
I20260928 14:45:47.868235 15828 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: No bootstrap required, opened a new log
I20260928 14:45:47.868378 15828 ts_tablet_manager.cc:1397] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 14:45:47.868866 15828 raft_consensus.cc:348] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.869016 15828 raft_consensus.cc:374] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.869068 15828 raft_consensus.cc:729] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226cec9290914bcd8a249c029b40030e, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.869223 15828 consensus_queue.cc:260] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.871225 15828 ts_tablet_manager.cc:1428] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:47.873147 15830 tablet_bootstrap.cc:492] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: No bootstrap required, opened a new log
I20260928 14:45:47.873258 15830 ts_tablet_manager.cc:1397] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260928 14:45:47.873773 15830 raft_consensus.cc:348] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.873910 15830 raft_consensus.cc:374] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.873942 15830 raft_consensus.cc:729] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a279907577a4281ae757d9f4015e0ce, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.874118 15830 consensus_queue.cc:260] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.874941 15830 ts_tablet_manager.cc:1428] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260928 14:45:47.885551 15836 raft_consensus.cc:482] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:47.885792 15836 raft_consensus.cc:504] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.887247 15836 leader_election.cc:288] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157), 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905)
I20260928 14:45:47.892426 15857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.887499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42157 (local address 127.13.223.66:43285)
0928 14:45:47.887904 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.887921 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.888032 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.888402 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.888406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.888423 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.888575 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.888583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.889846 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.889850 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.891112 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.891127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.891880 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.891883 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.892004 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.892034 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.892126 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.892233 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":29,"thread_start_us":106,"threads_started":1}
I20260928 14:45:47.892423 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.887826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:43285 (local address 127.13.223.65:42157)
0928 14:45:47.887961 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.887968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.887988 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:45:47.888169 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.888175 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.888240 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.888316 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.888755 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.889695 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.891288 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.891722 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.891870 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.891929 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.892185 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.892313 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:45:47.893085 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad0b58063494950a8d3fdb35b0e2c48" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" is_pre_election: true
I20260928 14:45:47.893267 15479 raft_consensus.cc:2454] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 0.
I20260928 14:45:47.893731 15550 leader_election.cc:302] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a279907577a4281ae757d9f4015e0ce, 6f4d052656cf4d4ead5e8dc5fec230be; no voters:
I20260928 14:45:47.893782 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.887853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44709 (local address 127.13.223.67:33905)
0928 14:45:47.887984 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.887988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.888269 (+ 281us) server_negotiation.cc:407] Connection header received
0928 14:45:47.888391 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.888394 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.888443 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.888523 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:47.888834 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.889430 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.891276 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.892031 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.892179 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.892238 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.893524 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.893632 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:45:47.894134 15836 raft_consensus.cc:2781] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:47.894234 15836 raft_consensus.cc:482] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:47.894274 15858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.887527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:33905 (local address 127.13.223.66:44709)
0928 14:45:47.888161 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.888176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.888283 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.888550 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.888553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.888566 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:47.888706 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.888716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.889550 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.889553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.891117 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.891127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.893090 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.893094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.893299 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.893329 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.893966 (+ 637us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.894063 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260928 14:45:47.894285 15836 raft_consensus.cc:3037] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:47.895430 15836 raft_consensus.cc:504] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.895879 15836 leader_election.cc:288] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 election: Requested vote from peers 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157), 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905)
I20260928 14:45:47.896275 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad0b58063494950a8d3fdb35b0e2c48" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
I20260928 14:45:47.896432 15479 raft_consensus.cc:3037] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:47.896598 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad0b58063494950a8d3fdb35b0e2c48" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226cec9290914bcd8a249c029b40030e"
I20260928 14:45:47.896605 15759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ad0b58063494950a8d3fdb35b0e2c48" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226cec9290914bcd8a249c029b40030e" is_pre_election: true
I20260928 14:45:47.896893 15759 raft_consensus.cc:2454] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 0.
I20260928 14:45:47.897588 15479 raft_consensus.cc:2454] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 1.
I20260928 14:45:47.897913 15550 leader_election.cc:302] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a279907577a4281ae757d9f4015e0ce, 6f4d052656cf4d4ead5e8dc5fec230be; no voters: 226cec9290914bcd8a249c029b40030e
I20260928 14:45:47.898155 15836 raft_consensus.cc:2781] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:47.898427 15836 raft_consensus.cc:686] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 1 LEADER]: Becoming Leader. State: Replica: 2a279907577a4281ae757d9f4015e0ce, State: Running, Role: LEADER
I20260928 14:45:47.898729 15836 consensus_queue.cc:237] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.901261 15320 catalog_manager.cc:5495] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a279907577a4281ae757d9f4015e0ce (127.13.223.66). New cstate: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:45:47 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606746, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:47.947053 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.936564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46691 (local address 127.0.0.1:34972)
0928 14:45:47.936894 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:47.936915 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:47.937022 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:47.937352 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:47.937358 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:47.937821 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:47.937934 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.937941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.939167 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.939170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.939756 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:47.939765 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.940414 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.940417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:47.940488 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.941154 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:47.941167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:47.943066 (+ 1899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:47.945284 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.945295 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.945301 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.945570 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.945882 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:47.945884 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:47.945885 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:47.945974 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:47.946433 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:47.946440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:47.946624 (+ 184us) client_negotiation.cc:769] Sending connection context
0928 14:45:47.946753 (+ 129us) client_negotiation.cc:240] Negotiation successful
0928 14:45:47.946903 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":126,"threads_started":1}
I20260928 14:45:47.947190 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:47.936649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.13.223.126:46691)
0928 14:45:47.936936 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:47.936948 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:47.936969 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:45:47.937129 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:47.937133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:47.937186 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:47.937301 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:47.938084 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.939000 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.939881 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:47.940275 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:47.940392 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:47.943262 (+ 2870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:47.943267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:47.943270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:47.943300 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:47.945138 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.945677 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.945681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.945683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.945783 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:47.946066 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:47.946069 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:47.946072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:47.946322 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:47.946427 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:47.946829 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 14:45:47.946985 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":109,"threads_started":1}
I20260928 14:45:47.952132 15320 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:34972:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:45:47.953032 15320 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:47.959136 15459 tablet_service.cc:1467] Processing CreateTablet for tablet 62c4f891dd494752828e02da38c8d732 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec87768618184117afe86e2657d8277b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:47.959414 15599 tablet_service.cc:1467] Processing CreateTablet for tablet 62c4f891dd494752828e02da38c8d732 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec87768618184117afe86e2657d8277b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:47.959668 15459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c4f891dd494752828e02da38c8d732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.959812 15599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c4f891dd494752828e02da38c8d732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.963049 15739 tablet_service.cc:1467] Processing CreateTablet for tablet 62c4f891dd494752828e02da38c8d732 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ec87768618184117afe86e2657d8277b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:47.963286 15830 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: Bootstrap starting.
I20260928 14:45:47.963511 15739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62c4f891dd494752828e02da38c8d732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:47.964389 15830 tablet_bootstrap.cc:654] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.965948 15829 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: Bootstrap starting.
I20260928 14:45:47.966403 15830 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: No bootstrap required, opened a new log
I20260928 14:45:47.966485 15830 ts_tablet_manager.cc:1397] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:45:47.966904 15828 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: Bootstrap starting.
I20260928 14:45:47.967005 15830 raft_consensus.cc:348] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.967133 15830 raft_consensus.cc:374] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.967168 15830 raft_consensus.cc:729] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a279907577a4281ae757d9f4015e0ce, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.967319 15830 consensus_queue.cc:260] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.967367 15829 tablet_bootstrap.cc:654] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.967774 15836 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:47.967916 15836 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:47.967981 15828 tablet_bootstrap.cc:654] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:47.968706 15830 ts_tablet_manager.cc:1428] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:47.969887 15829 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: No bootstrap required, opened a new log
I20260928 14:45:47.969998 15828 tablet_bootstrap.cc:492] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: No bootstrap required, opened a new log
I20260928 14:45:47.970011 15829 ts_tablet_manager.cc:1397] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:45:47.970079 15828 ts_tablet_manager.cc:1397] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:47.970537 15829 raft_consensus.cc:348] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.970611 15828 raft_consensus.cc:348] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.970700 15829 raft_consensus.cc:374] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.970749 15828 raft_consensus.cc:374] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:47.970762 15829 raft_consensus.cc:729] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4d052656cf4d4ead5e8dc5fec230be, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.970790 15828 raft_consensus.cc:729] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226cec9290914bcd8a249c029b40030e, State: Initialized, Role: FOLLOWER
I20260928 14:45:47.970947 15828 consensus_queue.cc:260] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.970978 15829 consensus_queue.cc:260] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:47.971274 15828 ts_tablet_manager.cc:1428] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:45:47.971217 15838 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:47.971431 15838 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:47.972415 15829 ts_tablet_manager.cc:1428] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:47.972349 15835 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:47.972577 15835 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:48.188838 15836 raft_consensus.cc:482] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:48.189041 15836 raft_consensus.cc:504] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:48.189700 15836 leader_election.cc:288] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157), 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905)
I20260928 14:45:48.190315 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c4f891dd494752828e02da38c8d732" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" is_pre_election: true
I20260928 14:45:48.190356 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c4f891dd494752828e02da38c8d732" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226cec9290914bcd8a249c029b40030e" is_pre_election: true
I20260928 14:45:48.190500 15479 raft_consensus.cc:2454] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 0.
I20260928 14:45:48.190537 15758 raft_consensus.cc:2454] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 0.
I20260928 14:45:48.190974 15550 leader_election.cc:302] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a279907577a4281ae757d9f4015e0ce, 6f4d052656cf4d4ead5e8dc5fec230be; no voters:
I20260928 14:45:48.191205 15836 raft_consensus.cc:2781] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:48.191291 15836 raft_consensus.cc:482] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:48.191350 15836 raft_consensus.cc:3037] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:48.192986 15836 raft_consensus.cc:504] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:48.193547 15836 leader_election.cc:288] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 election: Requested vote from peers 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157), 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905)
I20260928 14:45:48.193990 15758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c4f891dd494752828e02da38c8d732" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226cec9290914bcd8a249c029b40030e"
I20260928 14:45:48.194034 15479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "62c4f891dd494752828e02da38c8d732" candidate_uuid: "2a279907577a4281ae757d9f4015e0ce" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4d052656cf4d4ead5e8dc5fec230be"
I20260928 14:45:48.194157 15479 raft_consensus.cc:3037] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:48.194196 15758 raft_consensus.cc:3037] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:48.195425 15479 raft_consensus.cc:2454] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 1.
I20260928 14:45:48.195432 15758 raft_consensus.cc:2454] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a279907577a4281ae757d9f4015e0ce in term 1.
I20260928 14:45:48.195830 15550 leader_election.cc:302] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 226cec9290914bcd8a249c029b40030e, 2a279907577a4281ae757d9f4015e0ce, 6f4d052656cf4d4ead5e8dc5fec230be; no voters:
I20260928 14:45:48.196069 15836 raft_consensus.cc:2781] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:48.196278 15836 raft_consensus.cc:686] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [term 1 LEADER]: Becoming Leader. State: Replica: 2a279907577a4281ae757d9f4015e0ce, State: Running, Role: LEADER
I20260928 14:45:48.196455 15836 consensus_queue.cc:237] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } }
I20260928 14:45:48.197294 15859 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: TxnStatusTablet state changed. Reason: New leader 2a279907577a4281ae757d9f4015e0ce. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.197455 15859 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:48.197749 15872 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:45:48.197932 15872 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:45:48.199083 15322 catalog_manager.cc:5495] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a279907577a4281ae757d9f4015e0ce (127.13.223.66). New cstate: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:48.221776 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.217410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35098 (local address 127.13.223.66:37419)
0928 14:45:48.217565 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:48.217568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:48.217584 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:45:48.217742 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:48.217744 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:48.217803 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:48.217890 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:48.218140 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.218743 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.220047 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.220280 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.220401 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.220807 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:48.221343 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:48.221444 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:48.221586 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:45:48.221654 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:45:48.221844 15866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.217266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37419 (local address 127.0.0.1:35098)
0928 14:45:48.217444 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:48.217460 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:48.217575 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:48.217906 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:48.217910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:48.217917 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:48.218036 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:48.218042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.218863 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.218867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:48.219877 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:48.219890 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.220428 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.220432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:48.220540 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.220607 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:48.221465 (+ 858us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:48.221485 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:45:48.221564 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:45:48.221693 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:45:48.225726 15479 raft_consensus.cc:1261] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Refusing update from remote peer 2a279907577a4281ae757d9f4015e0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:48.225852 15758 raft_consensus.cc:1261] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Refusing update from remote peer 2a279907577a4281ae757d9f4015e0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:48.226316 15859 consensus_queue.cc:1035] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:48.226603 15836 consensus_queue.cc:1035] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:48.228654 15838 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: TxnStatusTablet state changed. Reason: New leader 2a279907577a4281ae757d9f4015e0ce. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.228799 15838 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:48.229435 15835 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: TxnStatusTablet state changed. Reason: New leader 2a279907577a4281ae757d9f4015e0ce. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.229560 15835 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:48.230393 15859 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.230508 15859 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:48.233171 15835 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.233320 15835 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 6f4d052656cf4d4ead5e8dc5fec230be: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:48.236498 15836 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.236693 15836 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 2a279907577a4281ae757d9f4015e0ce: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:48.238312 15841 mvcc.cc:205] Tried to move back new op lower bound from 7334325240725794816 to 7334325240614187008. Current Snapshot: MvccSnapshot[applied={T|T < 7334325240725794816}]
I20260928 14:45:48.238672 15838 tablet_replica.cc:439] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a279907577a4281ae757d9f4015e0ce" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 } } peers { permanent_uuid: "2a279907577a4281ae757d9f4015e0ce" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37419 } } peers { permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 } } }
I20260928 14:45:48.238803 15838 tablet_replica.cc:442] T 62c4f891dd494752828e02da38c8d732 P 226cec9290914bcd8a249c029b40030e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:48.243237 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.238101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35106 (local address 127.13.223.66:37419)
0928 14:45:48.238246 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:48.238253 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:48.238270 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:45:48.238315 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:48.238318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:48.238372 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:48.239476 (+ 1104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:48.239900 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.240606 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.241824 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.242080 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.242200 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.242515 (+ 315us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:48.242657 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:48.242751 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:48.242984 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 14:45:48.243065 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:45:48.256608 15823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.252343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:57911 (local address 127.13.223.66:37419)
0928 14:45:48.252545 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:48.252551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:48.252674 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:45:48.252800 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:48.252803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:48.252873 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:48.252975 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:48.253271 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.254145 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.255237 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.255626 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.255734 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.256046 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:48.256141 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:48.256220 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:48.256362 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:45:48.256418 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260928 14:45:48.256611 15880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.252247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37419 (local address 127.13.223.66:57911)
0928 14:45:48.252576 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:48.252591 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:48.252700 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:48.253017 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:48.253020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:48.253029 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:48.253154 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:48.253159 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.254276 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.254278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:48.255113 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:48.255123 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.255787 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.255790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:48.255865 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.255916 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:48.256251 (+ 335us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:48.256258 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:45:48.256328 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:45:48.256399 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":105,"threads_started":1}
I20260928 14:45:48.264581 15758 raft_consensus.cc:3037] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:48.264653 15479 raft_consensus.cc:1261] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Refusing update from remote peer 2a279907577a4281ae757d9f4015e0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:48.265087 15836 consensus_queue.cc:1035] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4d052656cf4d4ead5e8dc5fec230be" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:48.266285 15758 raft_consensus.cc:1261] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Refusing update from remote peer 2a279907577a4281ae757d9f4015e0ce: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:48.269866 15836 consensus_queue.cc:1035] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [LEADER]: Connected to new peer: Peer: permanent_uuid: "226cec9290914bcd8a249c029b40030e" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:48.323067 15322 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:48.323289 15322 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:48.325925 15322 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9253c621b33a41cd8abcbedda9a31c5c: Sending DeleteTablet for 3 replicas of tablet 9ad0b58063494950a8d3fdb35b0e2c48
I20260928 14:45:48.326727 15599 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:35084
I20260928 14:45:48.326766 15739 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39426
I20260928 14:45:48.326987 15852 tablet_replica.cc:330] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: stopping tablet replica
I20260928 14:45:48.327011 15459 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ad0b58063494950a8d3fdb35b0e2c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49282
I20260928 14:45:48.327148 15852 raft_consensus.cc:2229] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:48.327319 15852 raft_consensus.cc:2258] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:48.327535 15853 tablet_replica.cc:330] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: stopping tablet replica
I20260928 14:45:48.327672 15853 raft_consensus.cc:2229] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:48.327798 15853 raft_consensus.cc:2258] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:48.328733 15853 ts_tablet_manager.cc:1905] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:48.328791 15851 tablet_replica.cc:330] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: stopping tablet replica
I20260928 14:45:48.328922 15851 raft_consensus.cc:2229] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:45:48.329226 15851 raft_consensus.cc:2258] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:48.330021 15851 ts_tablet_manager.cc:1905] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:48.330272 15852 ts_tablet_manager.cc:1905] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:48.332254 15853 ts_tablet_manager.cc:1918] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:48.332389 15853 log.cc:1196] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/9ad0b58063494950a8d3fdb35b0e2c48
I20260928 14:45:48.332657 15853 ts_tablet_manager.cc:1939] T 9ad0b58063494950a8d3fdb35b0e2c48 P 6f4d052656cf4d4ead5e8dc5fec230be: Deleting consensus metadata
I20260928 14:45:48.333196 15851 ts_tablet_manager.cc:1918] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:48.333695 15306 catalog_manager.cc:4841] TS 6f4d052656cf4d4ead5e8dc5fec230be (127.13.223.65:42157): tablet 9ad0b58063494950a8d3fdb35b0e2c48 (table test-table [id=e96d6a4884304bdd837f4d047826d1e2]) successfully deleted
I20260928 14:45:48.333766 15851 log.cc:1196] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/9ad0b58063494950a8d3fdb35b0e2c48
I20260928 14:45:48.334103 15851 ts_tablet_manager.cc:1939] T 9ad0b58063494950a8d3fdb35b0e2c48 P 2a279907577a4281ae757d9f4015e0ce: Deleting consensus metadata
I20260928 14:45:48.334137 15852 ts_tablet_manager.cc:1918] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:48.334237 15852 log.cc:1196] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/9ad0b58063494950a8d3fdb35b0e2c48
I20260928 14:45:48.334570 15852 ts_tablet_manager.cc:1939] T 9ad0b58063494950a8d3fdb35b0e2c48 P 226cec9290914bcd8a249c029b40030e: Deleting consensus metadata
I20260928 14:45:48.335337 15309 catalog_manager.cc:4841] TS 2a279907577a4281ae757d9f4015e0ce (127.13.223.66:37419): tablet 9ad0b58063494950a8d3fdb35b0e2c48 (table test-table [id=e96d6a4884304bdd837f4d047826d1e2]) successfully deleted
I20260928 14:45:48.335474 15308 catalog_manager.cc:4841] TS 226cec9290914bcd8a249c029b40030e (127.13.223.67:33905): tablet 9ad0b58063494950a8d3fdb35b0e2c48 (table test-table [id=e96d6a4884304bdd837f4d047826d1e2]) successfully deleted
Sep 28 14:45:48 dist-test-slave-gkhx krb5kdc[15277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606747, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:45:48.353394 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.328976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.13.223.65:42157)
0928 14:45:48.329096 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:48.329100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:48.333805 (+ 4705us) server_negotiation.cc:407] Connection header received
0928 14:45:48.334105 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:48.334108 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:48.334175 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:48.334236 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:48.335485 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.336093 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.336986 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.337377 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.337514 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.347690 (+ 10176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:48.347710 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:48.347728 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:48.347773 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:48.350297 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:48.351027 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:48.351036 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:48.351038 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:48.351162 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:48.351728 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:48.351733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:48.351735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:48.352259 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:48.352535 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:48.353003 (+ 468us) server_negotiation.cc:299] Negotiation successful
0928 14:45:48.353183 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260928 14:45:48.354044 15502 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49288
I20260928 14:45:48.364908 15867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:48.356246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34984 (local address 127.13.223.126:46691)
0928 14:45:48.356398 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:48.356403 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:48.356459 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:45:48.356609 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:48.356613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:48.356670 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:48.356778 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:48.357492 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.358060 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.359074 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:48.359398 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:48.359539 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:48.361042 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:48.361046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:48.361054 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:48.361093 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:48.362772 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:48.363266 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:48.363272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:48.363280 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:48.363367 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:48.363675 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:48.363678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:48.363680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:48.363902 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:48.364168 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:48.364556 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 14:45:48.364716 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260928 14:45:48.365738 15322 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34984
I20260928 14:45:48.367547 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15394
I20260928 14:45:48.386843 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15534
I20260928 14:45:48.395721 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15674
I20260928 14:45:48.402922 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15290
2026-09-28T14:45:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3326 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:48 dist-test-slave-gkhx krb5kdc[15890](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:48 dist-test-slave-gkhx krb5kdc[15890](info): set up 2 sockets
Sep 28 14:45:48 dist-test-slave-gkhx krb5kdc[15890](info): commencing operation
krb5kdc: starting...
W20260928 14:45:49.541519 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.110s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:49 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:49.683527 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40313
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:40313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:49.803373 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:49.803742 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:49.803818 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:49.808137 15903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:49.808243 15903 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:45:49.808276 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:49.808323 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:49.808380 15903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:49.808416 15903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:49.814200 15903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:40313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:49.815443 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:49.816567 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:49.825989 15909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:49.826079 15908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:49.826053 15903 server_base.cc:1034] running on GCE node
W20260928 14:45:49.825994 15911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:49.827031 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:49.828236 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:49.829469 15903 hybrid_clock.cc:648] HybridClock initialized: now 1790606749829447 us; error 37 us; skew 500 ppm
Sep 28 14:45:49 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606749, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:49.839582 15903 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:49.840751 15903 webserver.cc:460] Webserver started at http://127.13.223.126:45389/ using document root <none> and password file <none>
I20260928 14:45:49.841369 15903 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:49.841439 15903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:49.841651 15903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:49.844348 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "ad975a927835412b98957441f95edf3c"
format_stamp: "Formatted at 2026-09-28 14:45:49 on dist-test-slave-gkhx"
server_key: "160fb4a7ca2c8720119ee10afb1df326"
server_key_iv: "fc8c06cbde3d59771d60334d8b623a37"
server_key_version: "encryptionkey@0"
I20260928 14:45:49.845227 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "ad975a927835412b98957441f95edf3c"
format_stamp: "Formatted at 2026-09-28 14:45:49 on dist-test-slave-gkhx"
server_key: "160fb4a7ca2c8720119ee10afb1df326"
server_key_iv: "fc8c06cbde3d59771d60334d8b623a37"
server_key_version: "encryptionkey@0"
I20260928 14:45:49.849686 15903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:45:49.852818 15918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:49.854238 15903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:45:49.854420 15903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "ad975a927835412b98957441f95edf3c"
format_stamp: "Formatted at 2026-09-28 14:45:49 on dist-test-slave-gkhx"
server_key: "160fb4a7ca2c8720119ee10afb1df326"
server_key_iv: "fc8c06cbde3d59771d60334d8b623a37"
server_key_version: "encryptionkey@0"
I20260928 14:45:49.854657 15903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:49.882278 15903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:49.886055 15903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:49.886293 15903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:49.895305 15903 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:40313
I20260928 14:45:49.895323 15980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:40313 every 8 connection(s)
I20260928 14:45:49.896364 15903 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:49.899452 15981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:49.905413 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Bootstrap starting.
I20260928 14:45:49.905859 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 15903
I20260928 14:45:49.905977 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:49.906250 14205 external_mini_cluster.cc:1428] Setting key 3c259e8de006ad0a3bb4cb20d137d90c
I20260928 14:45:49.908491 15981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:49.909313 15981 log.cc:824] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:49.911731 15981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: No bootstrap required, opened a new log
Sep 28 14:45:49 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:49.916666 15981 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } }
I20260928 14:45:49.916951 15981 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:49.917042 15981 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad975a927835412b98957441f95edf3c, State: Initialized, Role: FOLLOWER
I20260928 14:45:49.917569 15981 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } }
I20260928 14:45:49.917735 15981 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:49.917831 15981 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:49.917958 15981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:49.919036 15981 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } }
I20260928 14:45:49.919413 15981 leader_election.cc:302] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad975a927835412b98957441f95edf3c; no voters:
I20260928 14:45:49.919793 15981 leader_election.cc:288] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:49.919885 15986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:49.920065 15986 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [term 1 LEADER]: Becoming Leader. State: Replica: ad975a927835412b98957441f95edf3c, State: Running, Role: LEADER
I20260928 14:45:49.920389 15986 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } }
I20260928 14:45:49.921237 15981 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:49.922348 15988 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad975a927835412b98957441f95edf3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } } }
I20260928 14:45:49.922508 15988 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:49.922355 15987 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad975a927835412b98957441f95edf3c. Latest consensus state: current_term: 1 leader_uuid: "ad975a927835412b98957441f95edf3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad975a927835412b98957441f95edf3c" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40313 } } }
I20260928 14:45:49.922675 15987 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:49.922730 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:49.907739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59370 (local address 127.13.223.126:40313)
0928 14:45:49.908217 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:49.908234 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:49.908274 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:45:49.909073 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:49.909095 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:49.909534 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:49.909940 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:49.910818 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:49.911930 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:49.913128 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:49.913688 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:49.913819 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:49.916844 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:49.916850 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:49.916868 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:49.916900 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:49.919012 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:49.919534 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:49.919543 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:49.919551 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:49.919713 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:49.920083 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:49.920087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:49.920092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:49.920651 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:49.920897 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:49.921194 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:45:49.921548 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":166,"threads_started":1}
W20260928 14:45:49.926292 16001 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:49.926404 16001 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:49.926476 15995 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:49.927227 15995 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:49.934422 15995 catalog_manager.cc:1347] Generated new cluster ID: ce1783bfc1094a97989d7eef02787e8d
I20260928 14:45:49.934520 15995 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:49.944382 15995 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:49.945550 15995 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:49.959498 15995 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Generated new TSK 0
I20260928 14:45:49.960422 15995 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:45:49.998733 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40313
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:50.126152 16007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:50.126502 16007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:50.126621 16007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:50.131052 16007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:50.131146 16007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:50.131223 16007 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:45:50.131264 16007 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:50.131330 16007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:45:50.137190 16007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:50.138399 16007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:50.139559 16007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:50.150255 16012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:50.150255 16013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:50.150398 16007 server_base.cc:1034] running on GCE node
W20260928 14:45:50.150271 16015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:50.151046 16007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:50.151794 16007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:50.153096 16007 hybrid_clock.cc:648] HybridClock initialized: now 1790606750153051 us; error 65 us; skew 500 ppm
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:50.156464 16007 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:45:50.157532 16007 webserver.cc:460] Webserver started at http://127.13.223.65:36759/ using document root <none> and password file <none>
I20260928 14:45:50.158191 16007 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:50.158320 16007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:50.158516 16007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:50.160853 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "e114e8b4b5cd35197432e6b7e81262f9"
server_key_iv: "ce137b66314c832ac056acd0f939a7ae"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.161630 16007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "e114e8b4b5cd35197432e6b7e81262f9"
server_key_iv: "ce137b66314c832ac056acd0f939a7ae"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.166004 16007 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 14:45:50.169091 16022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.170715 16007 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:45:50.170840 16007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "e114e8b4b5cd35197432e6b7e81262f9"
server_key_iv: "ce137b66314c832ac056acd0f939a7ae"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.170957 16007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:50.197705 16007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:50.201515 16007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:50.201720 16007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:50.203344 16007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:50.203409 16007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.203457 16007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:50.203472 16007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.214897 16007 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:33219
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:50.214986 16137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:33219 every 8 connection(s)
I20260928 14:45:50.216063 16007 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:45:50.217178 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16007
I20260928 14:45:50.217315 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:45:50.217679 14205 external_mini_cluster.cc:1428] Setting key cb3ec29e9fe71f335e18cc9dc23848d3
I20260928 14:45:50.221511 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.204453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:34201 (local address 127.13.223.126:40313)
0928 14:45:50.204642 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.204647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.205581 (+ 934us) server_negotiation.cc:407] Connection header received
0928 14:45:50.207006 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.207010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.207074 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.207187 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.209164 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.210240 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.211948 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.212245 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.212391 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.215738 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.215743 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.215745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.215777 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.217873 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.218777 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.218781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.218783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.218897 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.219395 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.219399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.219401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.220019 (+ 618us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.220169 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.221134 (+ 965us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.221314 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:45:50.222805 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.204892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.65:34201)
0928 14:45:50.205401 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.205461 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.206688 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.207448 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.207458 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.208047 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.208907 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.208936 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.210409 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.210413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.211765 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.211780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.212449 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.212454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.212710 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.213517 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.213544 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.215468 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.218153 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.218160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.218175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.218644 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.219035 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.219038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.219040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.219256 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.220257 (+ 1001us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.220263 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.220788 (+ 525us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.221073 (+ 285us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.221491 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":159,"threads_started":1}
I20260928 14:45:50.224625 16138 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40313
I20260928 14:45:50.224808 16138 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:50.225519 16138 heartbeater.cc:511] Master 127.13.223.126:40313 requested a full tablet report, sending...
I20260928 14:45:50.227824 15934 ts_manager.cc:194] Registered new tserver with Master: dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219)
I20260928 14:45:50.229549 15934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:34201
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:45:50.238914 16145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.227732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.65:39031)
0928 14:45:50.228154 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.228170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.228301 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.228637 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.228640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.228905 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.229074 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.229083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.230435 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.230440 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.231877 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.232008 (+ 131us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.232732 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.232736 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.232828 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.233587 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.233602 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.234025 (+ 423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.236800 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.236808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.236813 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.237153 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.237518 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.237524 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.237529 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.237661 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.238231 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.238234 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.238342 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.238445 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.238729 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":136,"threads_started":1}
I20260928 14:45:50.238955 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.227807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:39031 (local address 127.13.223.126:40313)
0928 14:45:50.227963 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.227969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.228287 (+ 318us) server_negotiation.cc:407] Connection header received
0928 14:45:50.228406 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.228413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.228481 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.228587 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.229234 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.230224 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.232206 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.232550 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.232688 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.234341 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.234346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.234349 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.234381 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.236581 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.237300 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.237304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.237305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.237409 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.237793 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.237799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.237802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.238090 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.238216 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.238540 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.238730 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:45:50.255841 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40313
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:50.379273 16147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:50.379629 16147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:50.379714 16147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:50.384114 16147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:50.384248 16147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:50.384330 16147 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:45:50.384375 16147 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:50.384440 16147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:45:50.390242 16147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:50.391521 16147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:50.392648 16147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:50.404618 16153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:50.404733 16147 server_base.cc:1034] running on GCE node
W20260928 14:45:50.404606 16155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:50.404686 16152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:50.405651 16147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:50.406399 16147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:50.407570 16147 hybrid_clock.cc:648] HybridClock initialized: now 1790606750407532 us; error 52 us; skew 500 ppm
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:50.410797 16147 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:45:50.411845 16147 webserver.cc:460] Webserver started at http://127.13.223.66:33193/ using document root <none> and password file <none>
I20260928 14:45:50.412431 16147 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:50.412496 16147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:50.412679 16147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:50.415082 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "859c165582bb483fb16d3525cf9db61d"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "f2aecfa7b33cbfced25935e4b6b4d4cb"
server_key_iv: "ec59c18ffc3142ca6a974fb66f6f3cd5"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.415693 16147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "859c165582bb483fb16d3525cf9db61d"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "f2aecfa7b33cbfced25935e4b6b4d4cb"
server_key_iv: "ec59c18ffc3142ca6a974fb66f6f3cd5"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.419931 16147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:45:50.423009 16162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.424680 16147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:50.424836 16147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "859c165582bb483fb16d3525cf9db61d"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "f2aecfa7b33cbfced25935e4b6b4d4cb"
server_key_iv: "ec59c18ffc3142ca6a974fb66f6f3cd5"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.424978 16147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:50.447966 16147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:50.451649 16147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:50.451885 16147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:50.453620 16147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:50.453694 16147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.453756 16147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:50.453809 16147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:50.466141 16147 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:41643
I20260928 14:45:50.466269 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:41643 every 8 connection(s)
I20260928 14:45:50.467413 16147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:45:50.470525 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.454727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:51625 (local address 127.13.223.126:40313)
0928 14:45:50.454873 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.454876 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.456151 (+ 1275us) server_negotiation.cc:407] Connection header received
0928 14:45:50.457393 (+ 1242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.457397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.457462 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.457574 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.459662 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.460354 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.461879 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.462279 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.462406 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.465874 (+ 3468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.465879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.465882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.465926 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.467693 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.468545 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.468548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.468550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.468648 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.469194 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.469198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.469199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.469424 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.469516 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.470245 (+ 729us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.470374 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:45:50.471478 16171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.455305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.66:51625)
0928 14:45:50.455877 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.455924 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.457138 (+ 1214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.457766 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.457773 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.458377 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.459421 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.459452 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.460526 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.460529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.461745 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.461755 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.462470 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.462475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.462769 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.463528 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.463553 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.465601 (+ 2048us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.467836 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.467842 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.467859 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.468236 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.468788 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.468791 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.468792 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.469027 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.469573 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.469579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.469865 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.470126 (+ 261us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.470452 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":152,"threads_started":1}
I20260928 14:45:50.473137 16278 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40313
I20260928 14:45:50.473344 16278 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:50.473948 16278 heartbeater.cc:511] Master 127.13.223.126:40313 requested a full tablet report, sending...
I20260928 14:45:50.474066 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16147
I20260928 14:45:50.474200 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:45:50.474617 14205 external_mini_cluster.cc:1428] Setting key d884e58d991695e4f8731fce9c9efee1
I20260928 14:45:50.475404 15934 ts_manager.cc:194] Registered new tserver with Master: 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643)
I20260928 14:45:50.476243 15934 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:51625
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:45:50.486652 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.476223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:56307 (local address 127.13.223.126:40313)
0928 14:45:50.476373 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.476378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.478383 (+ 2005us) server_negotiation.cc:407] Connection header received
0928 14:45:50.478598 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.478604 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.478665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.478766 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.479299 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.480076 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.481143 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.481400 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.481533 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.483065 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.483069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.483071 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.483098 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.484733 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.485215 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.485219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.485221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.485308 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.485600 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.485604 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.485606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.485862 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.485977 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.486302 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.486430 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:45:50.486706 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.477918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.66:56307)
0928 14:45:50.478287 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.478306 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.478443 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.478786 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.478790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.479046 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.479161 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.479169 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.480253 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.480260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.480989 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.481000 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.481542 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.481546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.481671 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.482343 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.482357 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.482894 (+ 537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.484875 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.484880 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.484882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.485103 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.485406 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.485410 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.485412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.485503 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.485977 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.485980 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.486099 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.486255 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.486432 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:45:50.507354 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40313
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:50.633132 16287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:50.633455 16287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:50.633509 16287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:50.637964 16287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:50.638060 16287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:50.638108 16287 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 14:45:50.638123 16287 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:45:50.638157 16287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:45:50.643980 16287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:40313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:50.645145 16287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:50.646279 16287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:50.657224 16293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:50.657287 16292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:50.657233 16295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:50.657749 16287 server_base.cc:1034] running on GCE node
I20260928 14:45:50.658288 16287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:50.659070 16287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:50.660284 16287 hybrid_clock.cc:648] HybridClock initialized: now 1790606750660262 us; error 35 us; skew 500 ppm
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:50.663816 16287 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:45:50.664840 16287 webserver.cc:460] Webserver started at http://127.13.223.67:44729/ using document root <none> and password file <none>
I20260928 14:45:50.665527 16287 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:50.665607 16287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:50.665807 16287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:50.668498 16287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "b84731e5a2ac41de825a50df2e3b3817"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "ae4dde5c5d3db27c1a8c5b781572ba25"
server_key_iv: "0e6f65b2989f1b18dd756dbeb47a92fb"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.669175 16287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "b84731e5a2ac41de825a50df2e3b3817"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "ae4dde5c5d3db27c1a8c5b781572ba25"
server_key_iv: "0e6f65b2989f1b18dd756dbeb47a92fb"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.673959 16287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:45:50.677078 16302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.678884 16287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:50.679064 16287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "b84731e5a2ac41de825a50df2e3b3817"
format_stamp: "Formatted at 2026-09-28 14:45:50 on dist-test-slave-gkhx"
server_key: "ae4dde5c5d3db27c1a8c5b781572ba25"
server_key_iv: "0e6f65b2989f1b18dd756dbeb47a92fb"
server_key_version: "encryptionkey@0"
I20260928 14:45:50.679214 16287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:50.700665 16287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:50.704398 16287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:50.704634 16287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:50.706312 16287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:50.706388 16287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.706457 16287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:50.706511 16287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:50.717778 16287 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:42479
I20260928 14:45:50.717877 16417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:42479 every 8 connection(s)
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:50.718842 16287 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:45:50.723718 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.707095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41099 (local address 127.13.223.126:40313)
0928 14:45:50.707275 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.707279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.709171 (+ 1892us) server_negotiation.cc:407] Connection header received
0928 14:45:50.710623 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.710627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.710696 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.710816 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.712350 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.712971 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.714962 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.715236 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.715365 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.719064 (+ 3699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.719069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.719072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.719104 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.720724 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.721630 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.721634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.721635 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.721740 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.722320 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.722326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.722327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.722631 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.722730 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.723397 (+ 667us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.723529 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 14:45:50.724572 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.708313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.67:41099)
0928 14:45:50.708934 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.709028 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.710303 (+ 1275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.711036 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.711046 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.711531 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.712172 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.712188 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.713126 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.713132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.714801 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.714813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.715429 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.715436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.715722 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.716575 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.716608 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.718798 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.720964 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.720970 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.720985 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.721478 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.721902 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.721906 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.721908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.722146 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.722737 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.722743 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.723087 (+ 344us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.723310 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.723614 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260928 14:45:50.726298 16418 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40313
I20260928 14:45:50.726478 16418 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:50.727205 16418 heartbeater.cc:511] Master 127.13.223.126:40313 requested a full tablet report, sending...
I20260928 14:45:50.728572 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16287
I20260928 14:45:50.728582 15935 ts_manager.cc:194] Registered new tserver with Master: b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:50.728703 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:45:50.729068 14205 external_mini_cluster.cc:1428] Setting key 8467f4767717985630a671523f58900f
I20260928 14:45:50.729478 15935 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:41099
I20260928 14:45:50.731658 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:45:50.738757 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.729730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:52613 (local address 127.13.223.126:40313)
0928 14:45:50.730013 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.730018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.730128 (+ 110us) server_negotiation.cc:407] Connection header received
0928 14:45:50.730279 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.730282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.730338 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.730445 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.731071 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.731828 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.733177 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.733450 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.733550 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.735103 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.735108 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.735111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.735148 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.736675 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.737240 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.737243 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.737244 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.737332 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.737775 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.737779 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.737781 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.737983 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.738095 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.738391 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.738512 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198}
I20260928 14:45:50.738767 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.729636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.13.223.67:52613)
0928 14:45:50.730049 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.730065 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.730173 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.730465 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.730469 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.730770 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.730920 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.730930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.731976 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.731980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.733014 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.733030 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.733588 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.733595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.733744 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.734482 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.734505 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.734944 (+ 439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.736826 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.736831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.736833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.737074 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.737570 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.737574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.737576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.737673 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.738094 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.738097 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.738219 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.738332 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.738512 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":107,"threads_started":1}
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:50.763784 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.753928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.13.223.126:40313)
0928 14:45:50.754103 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.754110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.754829 (+ 719us) server_negotiation.cc:407] Connection header received
0928 14:45:50.755031 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.755034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.755095 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.755192 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.755960 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.756571 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.757487 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.757730 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.757845 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.760753 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.760756 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.760759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.760789 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.762204 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.762715 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.762718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.762719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.762797 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.763048 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.763051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.763052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.763240 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.763337 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.763554 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.763657 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:45:50.767078 15935 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:45:50.770143 15935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:50.786705 16438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.780255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42479 (local address 127.0.0.1:55196)
0928 14:45:50.781387 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.781399 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.781500 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.782032 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.782038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.782060 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:50.782217 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.782223 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783489 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.783493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.784952 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.784962 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.786062 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.786188 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786225 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.786339 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.786488 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260928 14:45:50.786756 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.780757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55196 (local address 127.13.223.67:42479)
0928 14:45:50.781261 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.781275 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.781500 (+ 225us) server_negotiation.cc:407] Connection header received
0928 14:45:50.781633 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.781640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.781862 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.782048 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:50.782346 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783333 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.785090 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.785917 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786045 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786243 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.786400 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.786515 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":131,"threads_started":1}
I20260928 14:45:50.786965 16435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.779841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.0.0.1:52890)
0928 14:45:50.781082 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.781153 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.781294 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.782238 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.782241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.782302 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:50.782419 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.782428 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783865 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.783868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.785320 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.785327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.786431 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.786618 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786664 (+ 46us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.786742 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.786825 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":222,"spinlock_wait_cycles":1280,"thread_start_us":104,"threads_started":1}
I20260928 14:45:50.787333 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.780612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38018 (local address 127.13.223.65:33219)
0928 14:45:50.781037 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.781048 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.781242 (+ 194us) server_negotiation.cc:407] Connection header received
0928 14:45:50.781775 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.781780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.781943 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.782075 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:50.782461 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783738 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.785318 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.786100 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786208 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786354 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.786760 (+ 406us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.787167 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":222,"thread_start_us":91,"threads_started":1}
I20260928 14:45:50.787401 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.780064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52890 (local address 127.13.223.66:41643)
0928 14:45:50.781490 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.781504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.781530 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:45:50.781636 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.781642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.781896 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.782075 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:50.782592 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783719 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.785447 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.786071 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786178 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786370 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.787113 (+ 743us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.787231 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1323,"thread_start_us":139,"threads_started":1}
I20260928 14:45:50.790016 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 (DEFAULT_TABLE table=test-table [id=41ee9a2c0a81406d9ff6865748405e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:50.790014 16352 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 (DEFAULT_TABLE table=test-table [id=41ee9a2c0a81406d9ff6865748405e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:50.790915 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.780068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33219 (local address 127.0.0.1:38018)
0928 14:45:50.781103 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.781153 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.781635 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.782217 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.782220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.782235 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:50.782342 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.782350 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.783865 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.783868 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.785184 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.785195 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.786420 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.786424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.786538 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.786592 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.790626 (+ 4034us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.790734 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"mutex_wait_us":26,"thread_start_us":127,"threads_started":1}
I20260928 14:45:50.791143 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5c54fd411a4877b45780510f7cd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.791159 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5c54fd411a4877b45780510f7cd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.794131 16072 tablet_service.cc:1467] Processing CreateTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 (DEFAULT_TABLE table=test-table [id=41ee9a2c0a81406d9ff6865748405e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:50.795478 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d5c54fd411a4877b45780510f7cd0b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.798507 16441 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Bootstrap starting.
I20260928 14:45:50.800239 16442 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Bootstrap starting.
I20260928 14:45:50.800920 16441 tablet_bootstrap.cc:654] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.801693 16441 log.cc:824] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:50.802639 16442 tablet_bootstrap.cc:654] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.803393 16444 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Bootstrap starting.
I20260928 14:45:50.803575 16442 log.cc:824] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:50.804642 16441 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: No bootstrap required, opened a new log
I20260928 14:45:50.804849 16441 ts_tablet_manager.cc:1397] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:45:50.805606 16442 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: No bootstrap required, opened a new log
I20260928 14:45:50.805814 16444 tablet_bootstrap.cc:654] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.805827 16442 ts_tablet_manager.cc:1397] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:45:50.806706 16444 log.cc:824] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:50.808841 16444 tablet_bootstrap.cc:492] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: No bootstrap required, opened a new log
I20260928 14:45:50.809033 16444 ts_tablet_manager.cc:1397] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260928 14:45:50.808843 16441 raft_consensus.cc:348] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.809124 16441 raft_consensus.cc:374] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.809190 16441 raft_consensus.cc:729] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.809180 16442 raft_consensus.cc:348] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.809477 16442 raft_consensus.cc:374] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.809540 16442 raft_consensus.cc:729] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.809960 16441 consensus_queue.cc:260] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.810158 16442 consensus_queue.cc:260] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.810897 16418 heartbeater.cc:503] Master 127.13.223.126:40313 was elected leader, sending a full tablet report...
I20260928 14:45:50.811153 16441 ts_tablet_manager.cc:1428] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:45:50.811589 16278 heartbeater.cc:503] Master 127.13.223.126:40313 was elected leader, sending a full tablet report...
I20260928 14:45:50.812112 16442 ts_tablet_manager.cc:1428] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260928 14:45:50.813748 16444 raft_consensus.cc:348] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.814011 16444 raft_consensus.cc:374] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.814091 16444 raft_consensus.cc:729] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.814621 16444 consensus_queue.cc:260] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.815400 16138 heartbeater.cc:503] Master 127.13.223.126:40313 was elected leader, sending a full tablet report...
I20260928 14:45:50.815662 16444 ts_tablet_manager.cc:1428] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:45:50.836556 16448 raft_consensus.cc:482] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:50.836836 16448 raft_consensus.cc:504] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.838125 16448 leader_election.cc:288] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219)
I20260928 14:45:50.842821 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.838683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:35721 (local address 127.13.223.65:33219)
0928 14:45:50.838804 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.838810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.839076 (+ 266us) server_negotiation.cc:407] Connection header received
0928 14:45:50.839225 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.839230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.839288 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.839378 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:50.839669 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.840260 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.841595 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.842118 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.842236 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.842289 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.842568 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.842674 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:45:50.842967 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.838651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:38333 (local address 127.13.223.66:41643)
0928 14:45:50.838811 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.838817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.838831 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:45:50.838899 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.838901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.838948 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.839060 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:50.839428 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.839996 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.841642 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.842262 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.842381 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.842430 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.842765 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.842850 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260928 14:45:50.842825 16450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.838469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33219 (local address 127.13.223.67:35721)
0928 14:45:50.838978 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.838990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.839105 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.839389 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.839391 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.839406 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:50.839557 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.839563 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.840382 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.840387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.841462 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.841473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.842246 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.842250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.842372 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.842398 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.842511 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.842630 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":274,"thread_start_us":133,"threads_started":1}
I20260928 14:45:50.842945 16312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.838470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.13.223.67:38333)
0928 14:45:50.838608 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.838625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.838732 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.839076 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.839079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.839183 (+ 104us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:50.839296 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.839302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.840131 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.840135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.841465 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.841477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.842492 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.842496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.842630 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.842650 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.842725 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.842786 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":4,"spinlock_wait_cycles":162688}
I20260928 14:45:50.844023 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5c54fd411a4877b45780510f7cd0b0" candidate_uuid: "b84731e5a2ac41de825a50df2e3b3817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" is_pre_election: true
I20260928 14:45:50.844046 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5c54fd411a4877b45780510f7cd0b0" candidate_uuid: "b84731e5a2ac41de825a50df2e3b3817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d" is_pre_election: true
I20260928 14:45:50.844368 16092 raft_consensus.cc:2454] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b84731e5a2ac41de825a50df2e3b3817 in term 0.
I20260928 14:45:50.844370 16232 raft_consensus.cc:2454] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b84731e5a2ac41de825a50df2e3b3817 in term 0.
I20260928 14:45:50.844959 16305 leader_election.cc:302] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, b84731e5a2ac41de825a50df2e3b3817, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:50.845270 16448 raft_consensus.cc:2781] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:50.845362 16448 raft_consensus.cc:482] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:50.845410 16448 raft_consensus.cc:3037] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:50.846526 16448 raft_consensus.cc:504] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.847081 16448 leader_election.cc:288] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [CANDIDATE]: Term 1 election: Requested vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219)
I20260928 14:45:50.847527 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5c54fd411a4877b45780510f7cd0b0" candidate_uuid: "b84731e5a2ac41de825a50df2e3b3817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
I20260928 14:45:50.847544 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d5c54fd411a4877b45780510f7cd0b0" candidate_uuid: "b84731e5a2ac41de825a50df2e3b3817" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d"
I20260928 14:45:50.847698 16232 raft_consensus.cc:3037] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:50.847661 16092 raft_consensus.cc:3037] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:50.848809 16232 raft_consensus.cc:2454] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b84731e5a2ac41de825a50df2e3b3817 in term 1.
I20260928 14:45:50.848814 16092 raft_consensus.cc:2454] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b84731e5a2ac41de825a50df2e3b3817 in term 1.
I20260928 14:45:50.849177 16305 leader_election.cc:302] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b84731e5a2ac41de825a50df2e3b3817, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:50.849390 16448 raft_consensus.cc:2781] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:50.849632 16448 raft_consensus.cc:686] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 1 LEADER]: Becoming Leader. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Running, Role: LEADER
I20260928 14:45:50.849999 16448 consensus_queue.cc:237] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.853890 15935 catalog_manager.cc:5495] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 reported cstate change: term changed from 0 to 1, leader changed from <none> to b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "b84731e5a2ac41de825a50df2e3b3817" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } health_report { overall_health: HEALTHY } } }
Sep 28 14:45:50 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606749, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:50.885756 15984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.876651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50108 (local address 127.13.223.126:40313)
0928 14:45:50.876786 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:50.876792 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:50.877029 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:45:50.877198 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:50.877203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:50.877244 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:50.877336 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:50.878054 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.878686 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.879522 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.879737 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.879848 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.882346 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:50.882349 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:50.882352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:50.882374 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:50.883930 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.884480 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.884483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.884484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.884559 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:50.884832 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:50.884834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:50.884835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:50.885084 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:50.885165 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:50.885471 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 14:45:50.885612 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":51}
I20260928 14:45:50.885807 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:50.876570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40313 (local address 127.0.0.1:50108)
0928 14:45:50.876875 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:50.876889 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:50.877086 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:50.877355 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:50.877358 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:50.877830 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:50.877926 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.877933 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.878808 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.878811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.879381 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:50.879389 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:50.879855 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:50.879858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:50.879979 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:50.880633 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:50.880645 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:50.882191 (+ 1546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:50.884119 (+ 1928us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.884132 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.884139 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.884389 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.884670 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:50.884672 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:50.884673 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:50.884755 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:50.885190 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:50.885196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:50.885315 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:45:50.885412 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 14:45:50.885552 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":20,"thread_start_us":105,"threads_started":1}
I20260928 14:45:50.890237 15935 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:50108:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:45:50.891088 15935 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:50.896440 16072 tablet_service.cc:1467] Processing CreateTablet for tablet d149ff7ae70042f78b5e359faff4925e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a70a45339c74307aeec34be6adc7f87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:50.896723 16200 tablet_service.cc:1467] Processing CreateTablet for tablet d149ff7ae70042f78b5e359faff4925e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a70a45339c74307aeec34be6adc7f87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:50.897311 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d149ff7ae70042f78b5e359faff4925e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.897298 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d149ff7ae70042f78b5e359faff4925e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.900465 16444 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: Bootstrap starting.
I20260928 14:45:50.900465 16441 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: Bootstrap starting.
I20260928 14:45:50.900519 16352 tablet_service.cc:1467] Processing CreateTablet for tablet d149ff7ae70042f78b5e359faff4925e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6a70a45339c74307aeec34be6adc7f87]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:45:50.901224 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d149ff7ae70042f78b5e359faff4925e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:50.901482 16441 tablet_bootstrap.cc:654] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.901482 16444 tablet_bootstrap.cc:654] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.903627 16441 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: No bootstrap required, opened a new log
I20260928 14:45:50.903726 16441 ts_tablet_manager.cc:1397] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:50.903908 16444 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: No bootstrap required, opened a new log
I20260928 14:45:50.903982 16444 ts_tablet_manager.cc:1397] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:45:50.904254 16441 raft_consensus.cc:348] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.904371 16441 raft_consensus.cc:374] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.904407 16441 raft_consensus.cc:729] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.904460 16444 raft_consensus.cc:348] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.904559 16444 raft_consensus.cc:374] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.904539 16441 consensus_queue.cc:260] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.904590 16444 raft_consensus.cc:729] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.904726 16444 consensus_queue.cc:260] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.904819 16441 ts_tablet_manager.cc:1428] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:45:50.904990 16444 ts_tablet_manager.cc:1428] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:45:50.905073 16447 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:50.905243 16447 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:50.905718 16442 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: Bootstrap starting.
I20260928 14:45:50.904992 16449 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:50.906189 16449 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:50.906922 16442 tablet_bootstrap.cc:654] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:50.908484 16442 tablet_bootstrap.cc:492] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: No bootstrap required, opened a new log
I20260928 14:45:50.908591 16442 ts_tablet_manager.cc:1397] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:50.908993 16442 raft_consensus.cc:348] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.909101 16442 raft_consensus.cc:374] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:50.909122 16442 raft_consensus.cc:729] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Initialized, Role: FOLLOWER
I20260928 14:45:50.909277 16442 consensus_queue.cc:260] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:50.909567 16442 ts_tablet_manager.cc:1428] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:45:50.909540 16448 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:50.909665 16448 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: This TxnStatusTablet replica's current role is: FOLLOWER
W20260928 14:45:50.967571 16139 tablet.cc:2369] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:50.968827 16279 tablet.cc:2369] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:45:50.970191 16419 tablet.cc:2369] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:45:51.088979 16447 raft_consensus.cc:482] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:51.089154 16447 raft_consensus.cc:504] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.090579 16447 leader_election.cc:288] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:51.096285 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.090997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:46243 (local address 127.13.223.67:42479)
0928 14:45:51.091137 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.091143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.091280 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:45:51.091368 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.091371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.091432 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.091513 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:51.091964 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.092814 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.094805 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.095460 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.095627 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.095684 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.096062 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.096150 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:45:51.096285 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.090882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:43557 (local address 127.13.223.65:33219)
0928 14:45:51.091019 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.091026 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.091198 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:45:51.091372 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.091375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.091424 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.091513 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:51.091964 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.092800 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.094807 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.095502 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.095627 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.095684 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.096062 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.096149 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:45:51.096293 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.090823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42479 (local address 127.13.223.66:46243)
0928 14:45:51.091193 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.091206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.091287 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.091550 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.091553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.091596 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:51.091785 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.091791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.093016 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.093020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.094642 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.094654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.095665 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.095670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.095844 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.095900 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.095983 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.096095 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":294,"spinlock_wait_cycles":7552,"thread_start_us":182,"threads_started":1}
I20260928 14:45:51.096307 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.090757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33219 (local address 127.13.223.66:43557)
0928 14:45:51.091088 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.091117 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.091240 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.091550 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.091553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.091595 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:51.091776 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.091785 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.093016 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.093020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.094640 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.094654 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.095665 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.095670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.095841 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.095892 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.095983 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.096095 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":229,"spinlock_wait_cycles":27264,"thread_start_us":91,"threads_started":1}
I20260928 14:45:51.097137 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d149ff7ae70042f78b5e359faff4925e" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" is_pre_election: true
I20260928 14:45:51.097328 16092 raft_consensus.cc:2454] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 0.
I20260928 14:45:51.097316 16372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d149ff7ae70042f78b5e359faff4925e" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817" is_pre_election: true
I20260928 14:45:51.097609 16372 raft_consensus.cc:2454] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 0.
I20260928 14:45:51.097754 16165 leader_election.cc:302] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:51.098129 16447 raft_consensus.cc:2781] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:51.098202 16447 raft_consensus.cc:482] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:51.098242 16447 raft_consensus.cc:3037] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.099287 16447 raft_consensus.cc:504] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.099655 16447 leader_election.cc:288] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 election: Requested vote from peers dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:51.100023 16372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d149ff7ae70042f78b5e359faff4925e" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817"
I20260928 14:45:51.100131 16372 raft_consensus.cc:3037] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.100157 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d149ff7ae70042f78b5e359faff4925e" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
I20260928 14:45:51.100334 16092 raft_consensus.cc:3037] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.101193 16372 raft_consensus.cc:2454] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 1.
I20260928 14:45:51.101428 16092 raft_consensus.cc:2454] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 1.
I20260928 14:45:51.101498 16166 leader_election.cc:302] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, b84731e5a2ac41de825a50df2e3b3817; no voters:
I20260928 14:45:51.101684 16447 raft_consensus.cc:2781] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:51.101873 16447 raft_consensus.cc:686] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [term 1 LEADER]: Becoming Leader. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Running, Role: LEADER
I20260928 14:45:51.102239 16447 consensus_queue.cc:237] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.104282 16464 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: TxnStatusTablet state changed. Reason: New leader 859c165582bb483fb16d3525cf9db61d. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.104414 16464 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:51.104730 16466 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:45:51.104948 16466 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:45:51.105818 15934 catalog_manager.cc:5495] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d reported cstate change: term changed from 0 to 1, leader changed from <none> to 859c165582bb483fb16d3525cf9db61d (127.13.223.66). New cstate: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:51.148115 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.143688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.0.0.1:52898)
0928 14:45:51.143839 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.143861 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.143985 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.144375 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.144377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.144385 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:51.144492 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.144496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.145537 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.145540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.146196 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.146206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.146799 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.146802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.146876 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.146923 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.147779 (+ 856us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.147795 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.147858 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.147953 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 14:45:51.148161 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.143821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52898 (local address 127.13.223.66:41643)
0928 14:45:51.143956 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.143960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.143976 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:45:51.144143 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.144146 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.144213 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.144326 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.144607 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.145422 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.146311 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.146690 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.146794 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.147058 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.147628 (+ 570us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.147736 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.147976 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.148047 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:45:51.153707 16092 raft_consensus.cc:1261] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Refusing update from remote peer 859c165582bb483fb16d3525cf9db61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.153707 16372 raft_consensus.cc:1261] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Refusing update from remote peer 859c165582bb483fb16d3525cf9db61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.154388 16464 consensus_queue.cc:1035] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:51.154585 16447 consensus_queue.cc:1035] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:51.157807 16448 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: TxnStatusTablet state changed. Reason: New leader 859c165582bb483fb16d3525cf9db61d. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.157954 16448 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:51.159178 16449 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: TxnStatusTablet state changed. Reason: New leader 859c165582bb483fb16d3525cf9db61d. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.159301 16449 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:51.161208 16464 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.161335 16464 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:51.161427 16469 mvcc.cc:205] Tried to move back new op lower bound from 7334325252718432256 to 7334325252517879808. Current Snapshot: MvccSnapshot[applied={T|T < 7334325252718432256}]
I20260928 14:45:51.162602 16471 mvcc.cc:205] Tried to move back new op lower bound from 7334325252718432256 to 7334325252517879808. Current Snapshot: MvccSnapshot[applied={T|T < 7334325252718432256}]
I20260928 14:45:51.162714 16448 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.162817 16448 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P b84731e5a2ac41de825a50df2e3b3817: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:51.161227 16447 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.162982 16447 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P 859c165582bb483fb16d3525cf9db61d: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:45:51.163478 16449 tablet_replica.cc:439] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } } }
I20260928 14:45:51.163606 16449 tablet_replica.cc:442] T d149ff7ae70042f78b5e359faff4925e P dd73379a7c2f41c9a03d32169c87d9dc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:45:51.166908 16470 mvcc.cc:205] Tried to move back new op lower bound from 7334325252718432256 to 7334325252517879808. Current Snapshot: MvccSnapshot[applied={T|T < 7334325252718432256 or (T in {7334325252718432256})}]
I20260928 14:45:51.173375 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.169191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.13.223.67:42479)
0928 14:45:51.169305 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.169310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.169324 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:45:51.169466 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.169469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.169535 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.169640 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.169865 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.170691 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.171570 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.171904 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.172010 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.172280 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.172856 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.172971 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.173123 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.173185 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:45:51.186067 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.182123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:59621 (local address 127.13.223.66:41643)
0928 14:45:51.182458 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.182462 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.182472 (+ 10us) server_negotiation.cc:407] Connection header received
0928 14:45:51.182516 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.182518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.182613 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.182681 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.183009 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.183569 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.184899 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.185152 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.185236 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.185635 (+ 399us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.185712 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.185763 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.185894 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.185945 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263}
I20260928 14:45:51.186133 16424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.182039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.13.223.67:59621)
0928 14:45:51.182226 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.182245 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.182327 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.182739 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.182743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.182752 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:51.182879 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.182886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.183682 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.183685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.184769 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.184782 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.185276 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.185280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.185403 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.185462 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.185795 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.185802 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.185877 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.185977 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 14:45:51.194303 16092 raft_consensus.cc:1261] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Refusing update from remote peer b84731e5a2ac41de825a50df2e3b3817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.194478 16232 raft_consensus.cc:1261] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Refusing update from remote peer b84731e5a2ac41de825a50df2e3b3817: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.194922 16448 consensus_queue.cc:1035] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:51.195130 16451 consensus_queue.cc:1035] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:51.225667 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.221251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:50845 (local address 127.13.223.67:42479)
0928 14:45:51.221395 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.221400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.221414 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:45:51.221559 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.221562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.221681 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.221757 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.222230 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.223054 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.224325 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.224636 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.224750 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.225037 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.225172 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.225244 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.225465 (+ 221us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.225530 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:51.226169 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.220845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42479 (local address 127.13.223.66:50845)
0928 14:45:51.221297 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.221315 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.221432 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.221915 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.221918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.221931 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:51.222056 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.222063 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.223239 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.223243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.224182 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.224228 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.224751 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.224754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.224841 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.224897 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.225340 (+ 443us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.225350 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.225894 (+ 544us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.226005 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":126,"threads_started":1}
I20260928 14:45:51.257205 15933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:51.257445 15933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:51.261072 15933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Sending DeleteTablet for 3 replicas of tablet 6d5c54fd411a4877b45780510f7cd0b0
I20260928 14:45:51.262064 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:51 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:52890
I20260928 14:45:51.262269 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:51 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38018
I20260928 14:45:51.262914 16488 tablet_replica.cc:330] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: stopping tablet replica
I20260928 14:45:51.263352 16488 raft_consensus.cc:2229] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:51.263630 16488 raft_consensus.cc:2258] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:51.264124 16352 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d5c54fd411a4877b45780510f7cd0b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:51 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:55196
I20260928 14:45:51.264740 15933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:45:51.265319 15933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:51.265761 16488 ts_tablet_manager.cc:1905] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:51.270027 16488 ts_tablet_manager.cc:1918] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:51.270257 16488 log.cc:1196] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/6d5c54fd411a4877b45780510f7cd0b0
I20260928 14:45:51.270740 16488 ts_tablet_manager.cc:1939] T 6d5c54fd411a4877b45780510f7cd0b0 P 859c165582bb483fb16d3525cf9db61d: Deleting consensus metadata
I20260928 14:45:51.271658 16200 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 (DEFAULT_TABLE table=test-table [id=d05984b861474ea682d4644dc74fb903]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:51.271973 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad576badd984e33ae14eeb2f5fcea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:51.272374 15920 catalog_manager.cc:4841] TS 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643): tablet 6d5c54fd411a4877b45780510f7cd0b0 (table test-table [id=41ee9a2c0a81406d9ff6865748405e73]) successfully deleted
I20260928 14:45:51.274710 16441 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Bootstrap starting.
I20260928 14:45:51.274822 16489 tablet_replica.cc:330] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: stopping tablet replica
I20260928 14:45:51.275219 16489 raft_consensus.cc:2229] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:51.275511 16489 raft_consensus.cc:2258] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:51.275544 16352 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 (DEFAULT_TABLE table=test-table [id=d05984b861474ea682d4644dc74fb903]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:51.275898 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad576badd984e33ae14eeb2f5fcea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:51.275950 16490 tablet_replica.cc:330] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: stopping tablet replica
I20260928 14:45:51.276244 16072 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 (DEFAULT_TABLE table=test-table [id=d05984b861474ea682d4644dc74fb903]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:51.276300 16490 raft_consensus.cc:2229] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:45:51.276513 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad576badd984e33ae14eeb2f5fcea87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:51.276713 16490 raft_consensus.cc:2258] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:51.278285 16489 ts_tablet_manager.cc:1905] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:51.278694 16442 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Bootstrap starting.
I20260928 14:45:51.279484 16441 tablet_bootstrap.cc:654] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:51.279748 16442 tablet_bootstrap.cc:654] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:51.280833 16444 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Bootstrap starting.
I20260928 14:45:51.281435 16489 ts_tablet_manager.cc:1918] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:51.281543 16489 log.cc:1196] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/6d5c54fd411a4877b45780510f7cd0b0
I20260928 14:45:51.281908 16489 ts_tablet_manager.cc:1939] T 6d5c54fd411a4877b45780510f7cd0b0 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting consensus metadata
I20260928 14:45:51.281991 16444 tablet_bootstrap.cc:654] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:51.283365 16444 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: No bootstrap required, opened a new log
I20260928 14:45:51.283352 15921 catalog_manager.cc:4841] TS dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219): tablet 6d5c54fd411a4877b45780510f7cd0b0 (table test-table [id=41ee9a2c0a81406d9ff6865748405e73]) successfully deleted
I20260928 14:45:51.283454 16444 ts_tablet_manager.cc:1397] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:45:51.283937 16444 raft_consensus.cc:348] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.284066 16444 raft_consensus.cc:374] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:51.284104 16444 raft_consensus.cc:729] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Initialized, Role: FOLLOWER
I20260928 14:45:51.284225 16444 consensus_queue.cc:260] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.284482 16444 ts_tablet_manager.cc:1428] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:45:51.286018 16490 ts_tablet_manager.cc:1905] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:51.287422 16442 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: No bootstrap required, opened a new log
I20260928 14:45:51.287516 16442 ts_tablet_manager.cc:1397] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 14:45:51.287618 16441 tablet_bootstrap.cc:492] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: No bootstrap required, opened a new log
I20260928 14:45:51.287721 16441 ts_tablet_manager.cc:1397] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 14:45:51.288270 16441 raft_consensus.cc:348] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.288453 16441 raft_consensus.cc:374] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:51.288503 16441 raft_consensus.cc:729] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Initialized, Role: FOLLOWER
I20260928 14:45:51.288666 16441 consensus_queue.cc:260] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.289011 16441 ts_tablet_manager.cc:1428] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:45:51.288010 16442 raft_consensus.cc:348] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.289505 16442 raft_consensus.cc:374] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:51.289570 16442 raft_consensus.cc:729] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Initialized, Role: FOLLOWER
I20260928 14:45:51.289707 16442 consensus_queue.cc:260] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.289986 16442 ts_tablet_manager.cc:1428] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:51.294708 16490 ts_tablet_manager.cc:1918] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:51.295092 16490 log.cc:1196] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/6d5c54fd411a4877b45780510f7cd0b0
I20260928 14:45:51.295514 16490 ts_tablet_manager.cc:1939] T 6d5c54fd411a4877b45780510f7cd0b0 P b84731e5a2ac41de825a50df2e3b3817: Deleting consensus metadata
I20260928 14:45:51.296977 15922 catalog_manager.cc:4841] TS b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479): tablet 6d5c54fd411a4877b45780510f7cd0b0 (table test-table [id=41ee9a2c0a81406d9ff6865748405e73]) successfully deleted
I20260928 14:45:51.317739 16449 raft_consensus.cc:482] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:51.318012 16449 raft_consensus.cc:504] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.319470 16449 leader_election.cc:288] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:51.319654 16475 raft_consensus.cc:482] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:51.319768 16475 raft_consensus.cc:504] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.320237 16475 leader_election.cc:288] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:51.320837 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" is_pre_election: true
I20260928 14:45:51.320996 16092 raft_consensus.cc:2454] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 0.
I20260928 14:45:51.321321 16165 leader_election.cc:302] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:51.321398 16372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817" is_pre_election: true
I20260928 14:45:51.321519 16475 raft_consensus.cc:2781] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:51.321554 16372 raft_consensus.cc:2454] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 0.
I20260928 14:45:51.321588 16475 raft_consensus.cc:482] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:51.321656 16475 raft_consensus.cc:3037] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.323179 16475 raft_consensus.cc:504] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.323625 16475 leader_election.cc:288] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 election: Requested vote from peers dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:51.324091 16372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817"
I20260928 14:45:51.324234 16372 raft_consensus.cc:3037] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.325608 16372 raft_consensus.cc:2454] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 1.
I20260928 14:45:51.326519 16092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "859c165582bb483fb16d3525cf9db61d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd73379a7c2f41c9a03d32169c87d9dc"
I20260928 14:45:51.326666 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.320063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:52095 (local address 127.13.223.66:41643)
0928 14:45:51.320188 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.320193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.320715 (+ 522us) server_negotiation.cc:407] Connection header received
0928 14:45:51.320790 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.320794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.320848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.320938 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:51.322285 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.323127 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.325186 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.325829 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.325953 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.326013 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.326423 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.326523 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:45:51.326666 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.320143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:43867 (local address 127.13.223.67:42479)
0928 14:45:51.320299 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.320304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.320811 (+ 507us) server_negotiation.cc:407] Connection header received
0928 14:45:51.321004 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.321010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.321065 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.321114 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:45:51.322940 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.323723 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.325271 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.325889 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.325967 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.326013 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.326381 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.326515 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 14:45:51.326719 16092 raft_consensus.cc:3037] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:51.327123 16495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.319728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42479 (local address 127.13.223.65:43867)
0928 14:45:51.320728 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.320742 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.320875 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.321259 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.321261 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.322658 (+ 1397us) spinlock_profiling.cc:243] Waited 994 us on lock 0x30062de8. stack: 00007f3f73530c78 00007f3f735311a9 00007f3f730f31d7 00000000005984a6 00000000005984f9 000000000059c488 00007f3f7352d2ab 00007f3f7352c9da 00007f3f7bc8e817 00007f3f7bca107f 00007f3f7bc97003 00007f3f77b85d12 00007f3f77b83934 00007f3f77bc4832 00007f3f77bc568c 00007f3f77bdb0d3
0928 14:45:51.322674 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:51.322797 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.322804 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.323944 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.323948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.325136 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.325154 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.326109 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.326112 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.326226 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.326254 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.326856 (+ 602us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.326963 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"spinlock_wait_cycles":2286336,"thread_start_us":119,"threads_started":1}
I20260928 14:45:51.327566 16494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.319707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.13.223.65:52095)
0928 14:45:51.320329 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.320347 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.320481 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.321089 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.321092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.321953 (+ 861us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:45:51.322137 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.322149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.323434 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.323437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.325021 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.325033 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.326186 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.326190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.326273 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.326307 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.327141 (+ 834us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.327429 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":74,"threads_started":1}
I20260928 14:45:51.327600 16372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817" is_pre_election: true
I20260928 14:45:51.327812 16372 raft_consensus.cc:2379] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd73379a7c2f41c9a03d32169c87d9dc in current term 1: Already voted for candidate 859c165582bb483fb16d3525cf9db61d in this term.
I20260928 14:45:51.328060 16092 raft_consensus.cc:2454] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 859c165582bb483fb16d3525cf9db61d in term 1.
I20260928 14:45:51.328512 16165 leader_election.cc:302] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:51.328487 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ad576badd984e33ae14eeb2f5fcea87" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d" is_pre_election: true
I20260928 14:45:51.328751 16232 raft_consensus.cc:2379] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dd73379a7c2f41c9a03d32169c87d9dc in current term 1: Already voted for candidate 859c165582bb483fb16d3525cf9db61d in this term.
I20260928 14:45:51.328850 16475 raft_consensus.cc:2781] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:51.329275 16475 raft_consensus.cc:686] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 LEADER]: Becoming Leader. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Running, Role: LEADER
I20260928 14:45:51.329804 16475 consensus_queue.cc:237] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:51.329988 16024 leader_election.cc:302] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dd73379a7c2f41c9a03d32169c87d9dc; no voters: 859c165582bb483fb16d3525cf9db61d, b84731e5a2ac41de825a50df2e3b3817
I20260928 14:45:51.330379 16449 raft_consensus.cc:2726] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 14:45:51.332581 15934 catalog_manager.cc:5495] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d reported cstate change: term changed from 0 to 1, leader changed from <none> to 859c165582bb483fb16d3525cf9db61d (127.13.223.66). New cstate: current_term: 1 leader_uuid: "859c165582bb483fb16d3525cf9db61d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:51.354914 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.351568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52912 (local address 127.13.223.66:41643)
0928 14:45:51.351715 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.351720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.351733 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:45:51.351778 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.351782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.351829 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.351912 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.352198 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.352846 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.353730 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.353972 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.354081 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.354369 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.354462 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.354545 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.354701 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.354764 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:45:51.366483 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.362582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.13.223.66:36745)
0928 14:45:51.362760 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:51.362777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:51.362896 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:51.363211 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:51.363214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:51.363221 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:51.363335 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.363341 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.364369 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.364372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.365105 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:51.365115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.365687 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.365689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:51.365802 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.365832 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.366152 (+ 320us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.366159 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:45:51.366231 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:45:51.366343 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":40}
I20260928 14:45:51.366508 16437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:51.362654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:36745 (local address 127.13.223.66:41643)
0928 14:45:51.362775 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:51.362780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:51.362833 (+ 53us) server_negotiation.cc:407] Connection header received
0928 14:45:51.363004 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:51.363008 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:51.363060 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:51.363146 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:51.363449 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.364244 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.365227 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:51.365570 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:51.365676 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:51.365949 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:51.366045 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:51.366119 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:51.366259 (+ 140us) server_negotiation.cc:299] Negotiation successful
0928 14:45:51.366339 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":49}
I20260928 14:45:51.373418 16372 raft_consensus.cc:1261] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Refusing update from remote peer 859c165582bb483fb16d3525cf9db61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.373441 16092 raft_consensus.cc:1261] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Refusing update from remote peer 859c165582bb483fb16d3525cf9db61d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:51.373819 16447 consensus_queue.cc:1035] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:51.373967 16475 consensus_queue.cc:1035] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:45:51.393296 16469 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:45:52.974699 15934 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:52.974936 15934 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:52.977957 15934 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Sending DeleteTablet for 3 replicas of tablet 6ad576badd984e33ae14eeb2f5fcea87
I20260928 14:45:52.978725 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:52 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38018
I20260928 14:45:52.978725 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:52 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:52890
I20260928 14:45:52.978824 16352 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad576badd984e33ae14eeb2f5fcea87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:52 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:55196
I20260928 14:45:52.979197 16520 tablet_replica.cc:330] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: stopping tablet replica
I20260928 14:45:52.979195 16521 tablet_replica.cc:330] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: stopping tablet replica
I20260928 14:45:52.979351 16521 raft_consensus.cc:2229] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:52.979362 16519 tablet_replica.cc:330] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: stopping tablet replica
I20260928 14:45:52.979367 16520 raft_consensus.cc:2229] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:52.979501 16521 raft_consensus.cc:2258] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:52.979501 16520 raft_consensus.cc:2258] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:52.979645 16519 raft_consensus.cc:2229] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:45:52.980049 16519 raft_consensus.cc:2258] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:52.980283 16521 ts_tablet_manager.cc:1905] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:52.980790 16519 ts_tablet_manager.cc:1905] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:52.982597 16520 ts_tablet_manager.cc:1905] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:52.983368 16521 ts_tablet_manager.cc:1918] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:45:52.983469 16521 log.cc:1196] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/6ad576badd984e33ae14eeb2f5fcea87
I20260928 14:45:52.983771 16521 ts_tablet_manager.cc:1939] T 6ad576badd984e33ae14eeb2f5fcea87 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting consensus metadata
I20260928 14:45:52.983927 16519 ts_tablet_manager.cc:1918] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:45:52.984031 16519 log.cc:1196] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/6ad576badd984e33ae14eeb2f5fcea87
I20260928 14:45:52.984344 16519 ts_tablet_manager.cc:1939] T 6ad576badd984e33ae14eeb2f5fcea87 P 859c165582bb483fb16d3525cf9db61d: Deleting consensus metadata
I20260928 14:45:52.984989 15921 catalog_manager.cc:4841] TS dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219): tablet 6ad576badd984e33ae14eeb2f5fcea87 (table test-table [id=d05984b861474ea682d4644dc74fb903]) successfully deleted
I20260928 14:45:52.985044 15920 catalog_manager.cc:4841] TS 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643): tablet 6ad576badd984e33ae14eeb2f5fcea87 (table test-table [id=d05984b861474ea682d4644dc74fb903]) successfully deleted
I20260928 14:45:52.986465 16520 ts_tablet_manager.cc:1918] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:45:52.986600 16520 log.cc:1196] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/6ad576badd984e33ae14eeb2f5fcea87
I20260928 14:45:52.986864 16520 ts_tablet_manager.cc:1939] T 6ad576badd984e33ae14eeb2f5fcea87 P b84731e5a2ac41de825a50df2e3b3817: Deleting consensus metadata
I20260928 14:45:52.988055 15922 catalog_manager.cc:4841] TS b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479): tablet 6ad576badd984e33ae14eeb2f5fcea87 (table test-table [id=d05984b861474ea682d4644dc74fb903]) successfully deleted
Sep 28 14:45:52 dist-test-slave-gkhx krb5kdc[15890](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:45:53 dist-test-slave-gkhx krb5kdc[15890](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:53.011180 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.000937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50110 (local address 127.13.223.126:40313)
0928 14:45:53.001369 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:53.001382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:53.001402 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:45:53.001471 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:53.001474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:53.001520 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:53.001678 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:53.002471 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.003198 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.004205 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.004444 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.004537 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.007511 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:53.007515 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:53.007518 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:53.007560 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:53.009104 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:53.009727 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:53.009730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:53.009731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:53.009821 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:53.010104 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:53.010106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:53.010107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:53.010316 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:53.010439 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:53.010835 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 14:45:53.011035 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":143,"threads_started":1}
I20260928 14:45:53.014695 15934 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:45:53.015146 15934 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:53.020660 16072 tablet_service.cc:1467] Processing CreateTablet for tablet b891ca2267a6404181755f61b251ff41 (DEFAULT_TABLE table=test-table [id=792d56289feb473090a3ffb20312c613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.020889 16352 tablet_service.cc:1467] Processing CreateTablet for tablet b891ca2267a6404181755f61b251ff41 (DEFAULT_TABLE table=test-table [id=792d56289feb473090a3ffb20312c613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.020911 16200 tablet_service.cc:1467] Processing CreateTablet for tablet b891ca2267a6404181755f61b251ff41 (DEFAULT_TABLE table=test-table [id=792d56289feb473090a3ffb20312c613]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.021020 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b891ca2267a6404181755f61b251ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.021140 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b891ca2267a6404181755f61b251ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.021224 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b891ca2267a6404181755f61b251ff41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.024106 16531 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Bootstrap starting.
I20260928 14:45:53.024396 16532 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Bootstrap starting.
I20260928 14:45:53.025149 16531 tablet_bootstrap.cc:654] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.025219 16532 tablet_bootstrap.cc:654] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.025960 16530 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Bootstrap starting.
I20260928 14:45:53.026648 16532 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: No bootstrap required, opened a new log
I20260928 14:45:53.026855 16531 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: No bootstrap required, opened a new log
I20260928 14:45:53.026954 16531 ts_tablet_manager.cc:1397] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:53.027037 16530 tablet_bootstrap.cc:654] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.027377 16532 ts_tablet_manager.cc:1397] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:53.027493 16531 raft_consensus.cc:348] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.027628 16531 raft_consensus.cc:374] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.027662 16531 raft_consensus.cc:729] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.027829 16531 consensus_queue.cc:260] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.027850 16532 raft_consensus.cc:348] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.027966 16532 raft_consensus.cc:374] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.028019 16532 raft_consensus.cc:729] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.028172 16532 consensus_queue.cc:260] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.028365 16531 ts_tablet_manager.cc:1428] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:45:53.028501 16530 tablet_bootstrap.cc:492] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: No bootstrap required, opened a new log
I20260928 14:45:53.028594 16530 ts_tablet_manager.cc:1397] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:53.029053 16530 raft_consensus.cc:348] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.029186 16530 raft_consensus.cc:374] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.029219 16530 raft_consensus.cc:729] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.029253 16532 ts_tablet_manager.cc:1428] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:53.029347 16530 consensus_queue.cc:260] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.029738 16530 ts_tablet_manager.cc:1428] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:45:53.262735 16536 raft_consensus.cc:482] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:53.262882 16536 raft_consensus.cc:504] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.263408 16536 leader_election.cc:288] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:53.263799 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b891ca2267a6404181755f61b251ff41" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d" is_pre_election: true
I20260928 14:45:53.263967 16232 raft_consensus.cc:2454] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 0.
I20260928 14:45:53.263943 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b891ca2267a6404181755f61b251ff41" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817" is_pre_election: true
I20260928 14:45:53.264113 16371 raft_consensus.cc:2454] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 0.
I20260928 14:45:53.264387 16024 leader_election.cc:302] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:53.264601 16536 raft_consensus.cc:2781] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:53.264667 16536 raft_consensus.cc:482] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:53.264717 16536 raft_consensus.cc:3037] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.265836 16536 raft_consensus.cc:504] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.266459 16536 leader_election.cc:288] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 election: Requested vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:53.266937 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b891ca2267a6404181755f61b251ff41" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817"
I20260928 14:45:53.267107 16371 raft_consensus.cc:3037] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.267416 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b891ca2267a6404181755f61b251ff41" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d"
I20260928 14:45:53.267688 16232 raft_consensus.cc:3037] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.268599 16371 raft_consensus.cc:2454] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 1.
I20260928 14:45:53.269222 16232 raft_consensus.cc:2454] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 1.
I20260928 14:45:53.269321 16026 leader_election.cc:302] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b84731e5a2ac41de825a50df2e3b3817, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:53.269578 16536 raft_consensus.cc:2781] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:53.269843 16536 raft_consensus.cc:686] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 LEADER]: Becoming Leader. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Running, Role: LEADER
I20260928 14:45:53.270184 16536 consensus_queue.cc:237] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.273641 15935 catalog_manager.cc:5495] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc reported cstate change: term changed from 0 to 1, leader changed from <none> to dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65). New cstate: current_term: 1 leader_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:53.336035 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.331684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.13.223.65:33219)
0928 14:45:53.331956 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:53.331963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:53.331976 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:45:53.332126 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:53.332128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:53.332186 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:53.332298 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:53.332611 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.333227 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.334219 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.334466 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.334624 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.334901 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:53.335456 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:53.335562 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:53.335711 (+ 149us) server_negotiation.cc:299] Negotiation successful
0928 14:45:53.335768 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":110,"threads_started":1}
I20260928 14:45:53.348796 16544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.344540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41643 (local address 127.13.223.65:46257)
0928 14:45:53.345071 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:53.345099 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:53.345214 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:53.345480 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:53.345482 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:53.345490 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:53.345635 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:53.345640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.346471 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.346474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:53.347207 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:53.347217 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.347843 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.347846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:53.348000 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.348057 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:53.348474 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:53.348482 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:45:53.348552 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:45:53.348641 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":139,"threads_started":1}
I20260928 14:45:53.348784 16545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.344873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:46257 (local address 127.13.223.66:41643)
0928 14:45:53.345232 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:53.345245 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:53.345259 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:45:53.345308 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:53.345310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:53.345355 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:53.345432 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:53.345749 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.346343 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.347434 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.347695 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.347789 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.348229 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:53.348321 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:53.348392 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:53.348584 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 14:45:53.348647 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":127,"threads_started":1}
I20260928 14:45:53.355810 16232 raft_consensus.cc:1261] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Refusing update from remote peer dd73379a7c2f41c9a03d32169c87d9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:53.355933 16371 raft_consensus.cc:1261] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Refusing update from remote peer dd73379a7c2f41c9a03d32169c87d9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:53.356416 16536 consensus_queue.cc:1035] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:53.356649 16538 consensus_queue.cc:1035] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:53.376881 16550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.372740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33219 (local address 127.13.223.66:56359)
0928 14:45:53.373088 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:53.373107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:53.373205 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:53.373525 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:53.373527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:53.373533 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:45:53.373691 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:53.373697 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.374677 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.374679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:53.375442 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:53.375452 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.376080 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.376085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:53.376164 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.376222 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:45:53.376540 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:45:53.376546 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:45:53.376605 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:45:53.376712 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":110,"threads_started":1}
I20260928 14:45:53.376878 16541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:53.373012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:56359 (local address 127.13.223.65:33219)
0928 14:45:53.373168 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:53.373172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:53.373209 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:45:53.373308 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:53.373311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:53.373372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:53.373520 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:45:53.373820 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.374533 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.375577 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:53.375872 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:53.375982 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:53.376334 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:45:53.376419 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:45:53.376528 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:53.376633 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:45:53.376705 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:45:53.405370 15933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:53.405543 15933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:53.408229 15933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Sending DeleteTablet for 3 replicas of tablet b891ca2267a6404181755f61b251ff41
I20260928 14:45:53.409015 16352 tablet_service.cc:1514] Processing DeleteTablet for tablet b891ca2267a6404181755f61b251ff41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:53 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:55196
I20260928 14:45:53.409085 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet b891ca2267a6404181755f61b251ff41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:53 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38018
I20260928 14:45:53.409193 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet b891ca2267a6404181755f61b251ff41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:53 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:52890
I20260928 14:45:53.409262 16521 tablet_replica.cc:330] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: stopping tablet replica
I20260928 14:45:53.409263 16520 tablet_replica.cc:330] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: stopping tablet replica
I20260928 14:45:53.409379 16521 raft_consensus.cc:2229] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:45:53.409379 16520 raft_consensus.cc:2229] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:53.409389 16519 tablet_replica.cc:330] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: stopping tablet replica
I20260928 14:45:53.409508 16520 raft_consensus.cc:2258] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:53.409510 16519 raft_consensus.cc:2229] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:53.409626 16519 raft_consensus.cc:2258] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:53.409683 16521 raft_consensus.cc:2258] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:53.410360 16520 ts_tablet_manager.cc:1905] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:53.410696 16519 ts_tablet_manager.cc:1905] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:53.410902 16521 ts_tablet_manager.cc:1905] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:53.413206 16519 ts_tablet_manager.cc:1918] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:53.413206 16520 ts_tablet_manager.cc:1918] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:53.413285 16519 log.cc:1196] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/b891ca2267a6404181755f61b251ff41
I20260928 14:45:53.413285 16520 log.cc:1196] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/b891ca2267a6404181755f61b251ff41
I20260928 14:45:53.413555 16519 ts_tablet_manager.cc:1939] T b891ca2267a6404181755f61b251ff41 P 859c165582bb483fb16d3525cf9db61d: Deleting consensus metadata
I20260928 14:45:53.413561 16520 ts_tablet_manager.cc:1939] T b891ca2267a6404181755f61b251ff41 P b84731e5a2ac41de825a50df2e3b3817: Deleting consensus metadata
I20260928 14:45:53.413652 16521 ts_tablet_manager.cc:1918] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:45:53.413726 16521 log.cc:1196] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/b891ca2267a6404181755f61b251ff41
I20260928 14:45:53.413982 16521 ts_tablet_manager.cc:1939] T b891ca2267a6404181755f61b251ff41 P dd73379a7c2f41c9a03d32169c87d9dc: Deleting consensus metadata
I20260928 14:45:53.414664 15922 catalog_manager.cc:4841] TS b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479): tablet b891ca2267a6404181755f61b251ff41 (table test-table [id=792d56289feb473090a3ffb20312c613]) successfully deleted
I20260928 14:45:53.414723 15921 catalog_manager.cc:4841] TS dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219): tablet b891ca2267a6404181755f61b251ff41 (table test-table [id=792d56289feb473090a3ffb20312c613]) successfully deleted
I20260928 14:45:53.415189 15933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:45:53.415560 15920 catalog_manager.cc:4841] TS 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643): tablet b891ca2267a6404181755f61b251ff41 (table test-table [id=792d56289feb473090a3ffb20312c613]) successfully deleted
W20260928 14:45:53.415663 15933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:45:53.421562 16072 tablet_service.cc:1467] Processing CreateTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f (DEFAULT_TABLE table=test-table [id=dd8114cc07e54470a2748387e13ca41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.421890 16200 tablet_service.cc:1467] Processing CreateTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f (DEFAULT_TABLE table=test-table [id=dd8114cc07e54470a2748387e13ca41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.421999 16072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf2f755aace47c3a97f701c7d6a2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.422282 16200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf2f755aace47c3a97f701c7d6a2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.424769 16531 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc: Bootstrap starting.
I20260928 14:45:53.425036 16352 tablet_service.cc:1467] Processing CreateTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f (DEFAULT_TABLE table=test-table [id=dd8114cc07e54470a2748387e13ca41e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:45:53.425361 16352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acf2f755aace47c3a97f701c7d6a2c4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:53.426191 16531 tablet_bootstrap.cc:654] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.427765 16531 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc: No bootstrap required, opened a new log
I20260928 14:45:53.427912 16531 ts_tablet_manager.cc:1397] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20260928 14:45:53.428510 16530 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Bootstrap starting.
I20260928 14:45:53.428520 16531 raft_consensus.cc:348] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.428689 16531 raft_consensus.cc:374] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.428758 16531 raft_consensus.cc:729] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.429244 16532 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Bootstrap starting.
I20260928 14:45:53.429499 16530 tablet_bootstrap.cc:654] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.429590 16531 consensus_queue.cc:260] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.430007 16531 ts_tablet_manager.cc:1428] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:45:53.430383 16532 tablet_bootstrap.cc:654] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:53.431890 16532 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: No bootstrap required, opened a new log
I20260928 14:45:53.432020 16532 ts_tablet_manager.cc:1397] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:53.432649 16530 tablet_bootstrap.cc:492] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: No bootstrap required, opened a new log
I20260928 14:45:53.432600 16532 raft_consensus.cc:348] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.432749 16530 ts_tablet_manager.cc:1397] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 14:45:53.432747 16532 raft_consensus.cc:374] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.432792 16532 raft_consensus.cc:729] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 859c165582bb483fb16d3525cf9db61d, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.432948 16532 consensus_queue.cc:260] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.433202 16530 raft_consensus.cc:348] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.433280 16532 ts_tablet_manager.cc:1428] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:45:53.433312 16530 raft_consensus.cc:374] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:53.433344 16530 raft_consensus.cc:729] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84731e5a2ac41de825a50df2e3b3817, State: Initialized, Role: FOLLOWER
I20260928 14:45:53.433463 16530 consensus_queue.cc:260] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.433728 16530 ts_tablet_manager.cc:1428] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:45:53.684335 16551 raft_consensus.cc:482] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:45:53.684517 16551 raft_consensus.cc:504] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.685127 16551 leader_election.cc:288] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:53.685618 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf2f755aace47c3a97f701c7d6a2c4f" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d" is_pre_election: true
I20260928 14:45:53.685638 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf2f755aace47c3a97f701c7d6a2c4f" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817" is_pre_election: true
I20260928 14:45:53.685808 16232 raft_consensus.cc:2454] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 0.
I20260928 14:45:53.685820 16371 raft_consensus.cc:2454] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 0.
I20260928 14:45:53.686187 16024 leader_election.cc:302] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 859c165582bb483fb16d3525cf9db61d, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:53.686462 16551 raft_consensus.cc:2781] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:45:53.686530 16551 raft_consensus.cc:482] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:45:53.686633 16551 raft_consensus.cc:3037] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.687803 16551 raft_consensus.cc:504] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.688274 16551 leader_election.cc:288] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 election: Requested vote from peers 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643), b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479)
I20260928 14:45:53.688726 16232 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf2f755aace47c3a97f701c7d6a2c4f" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "859c165582bb483fb16d3525cf9db61d"
I20260928 14:45:53.688751 16371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acf2f755aace47c3a97f701c7d6a2c4f" candidate_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84731e5a2ac41de825a50df2e3b3817"
I20260928 14:45:53.688877 16232 raft_consensus.cc:3037] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.688896 16371 raft_consensus.cc:3037] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:53.690016 16371 raft_consensus.cc:2454] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 1.
I20260928 14:45:53.690017 16232 raft_consensus.cc:2454] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd73379a7c2f41c9a03d32169c87d9dc in term 1.
I20260928 14:45:53.690392 16026 leader_election.cc:302] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b84731e5a2ac41de825a50df2e3b3817, dd73379a7c2f41c9a03d32169c87d9dc; no voters:
I20260928 14:45:53.690656 16551 raft_consensus.cc:2781] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:53.690789 16551 raft_consensus.cc:686] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [term 1 LEADER]: Becoming Leader. State: Replica: dd73379a7c2f41c9a03d32169c87d9dc, State: Running, Role: LEADER
I20260928 14:45:53.690970 16551 consensus_queue.cc:237] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } }
I20260928 14:45:53.693073 15933 catalog_manager.cc:5495] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc reported cstate change: term changed from 0 to 1, leader changed from <none> to dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65). New cstate: current_term: 1 leader_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dd73379a7c2f41c9a03d32169c87d9dc" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 } health_report { overall_health: UNKNOWN } } }
I20260928 14:45:53.747284 16371 raft_consensus.cc:1261] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Refusing update from remote peer dd73379a7c2f41c9a03d32169c87d9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:53.747702 16232 raft_consensus.cc:1261] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Refusing update from remote peer dd73379a7c2f41c9a03d32169c87d9dc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:45:53.747750 16536 consensus_queue.cc:1035] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "b84731e5a2ac41de825a50df2e3b3817" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42479 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:45:53.748103 16536 consensus_queue.cc:1035] T acf2f755aace47c3a97f701c7d6a2c4f P dd73379a7c2f41c9a03d32169c87d9dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "859c165582bb483fb16d3525cf9db61d" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41643 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:45:53.761816 16504 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 14:45:55.319947 15933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:55.320123 15933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50110:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:45:55.323196 15933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ad975a927835412b98957441f95edf3c: Sending DeleteTablet for 3 replicas of tablet acf2f755aace47c3a97f701c7d6a2c4f
I20260928 14:45:55.324167 16352 tablet_service.cc:1514] Processing DeleteTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:55 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:55196
I20260928 14:45:55.324283 16200 tablet_service.cc:1514] Processing DeleteTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:55 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:52890
I20260928 14:45:55.324569 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16007
I20260928 14:45:55.324630 16072 tablet_service.cc:1514] Processing DeleteTablet for tablet acf2f755aace47c3a97f701c7d6a2c4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:45:55 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38018
I20260928 14:45:55.326166 16581 tablet_replica.cc:330] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: stopping tablet replica
I20260928 14:45:55.326368 16581 raft_consensus.cc:2229] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:55.326576 16581 raft_consensus.cc:2258] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:55.327173 16582 tablet_replica.cc:330] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: stopping tablet replica
I20260928 14:45:55.327325 16582 raft_consensus.cc:2229] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:45:55.327478 16582 raft_consensus.cc:2258] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:45:55.327522 16581 ts_tablet_manager.cc:1905] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:55.328109 16582 ts_tablet_manager.cc:1905] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:45:55.330695 16582 ts_tablet_manager.cc:1918] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:45:55.330796 16582 log.cc:1196] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/acf2f755aace47c3a97f701c7d6a2c4f
I20260928 14:45:55.331081 16582 ts_tablet_manager.cc:1939] T acf2f755aace47c3a97f701c7d6a2c4f P 859c165582bb483fb16d3525cf9db61d: Deleting consensus metadata
I20260928 14:45:55.331348 16581 ts_tablet_manager.cc:1918] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 14:45:55.331446 16581 log.cc:1196] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/acf2f755aace47c3a97f701c7d6a2c4f
I20260928 14:45:55.331758 16581 ts_tablet_manager.cc:1939] T acf2f755aace47c3a97f701c7d6a2c4f P b84731e5a2ac41de825a50df2e3b3817: Deleting consensus metadata
I20260928 14:45:55.332034 15920 catalog_manager.cc:4841] TS 859c165582bb483fb16d3525cf9db61d (127.13.223.66:41643): tablet acf2f755aace47c3a97f701c7d6a2c4f (table test-table [id=dd8114cc07e54470a2748387e13ca41e]) successfully deleted
I20260928 14:45:55.332674 15922 catalog_manager.cc:4841] TS b84731e5a2ac41de825a50df2e3b3817 (127.13.223.67:42479): tablet acf2f755aace47c3a97f701c7d6a2c4f (table test-table [id=dd8114cc07e54470a2748387e13ca41e]) successfully deleted
W20260928 14:45:55.334184 15921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:33219 (error 108)
I20260928 14:45:55.334575 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16147
I20260928 14:45:55.336472 16584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:55.335834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33219 (local address 127.0.0.1:38034)
0928 14:45:55.336264 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:55.336369 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33219: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":128,"threads_started":1}
W20260928 14:45:55.336673 15921 catalog_manager.cc:4584] TS dd73379a7c2f41c9a03d32169c87d9dc (127.13.223.65:33219): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet acf2f755aace47c3a97f701c7d6a2c4f: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33219: connect: Connection refused (error 111)
I20260928 14:45:55.344348 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16287
I20260928 14:45:55.352080 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 15903
2026-09-28T14:45:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6947 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:55 dist-test-slave-gkhx krb5kdc[16587](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:55 dist-test-slave-gkhx krb5kdc[16587](info): set up 2 sockets
Sep 28 14:45:55 dist-test-slave-gkhx krb5kdc[16587](info): commencing operation
krb5kdc: starting...
W20260928 14:45:56.484668 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:56 dist-test-slave-gkhx krb5kdc[16587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:56.618116 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34933
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34933
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:56.733631 16600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:56.734023 16600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:56.734110 16600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:56.738410 16600 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:56.738574 16600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:56.738643 16600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:56.738703 16600 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:56.738758 16600 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:56.744787 16600 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34933
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:56.746016 16600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:56.747148 16600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:56.756829 16600 server_base.cc:1034] running on GCE node
W20260928 14:45:56.756769 16605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:56.756774 16608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:56.756794 16606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:56.757759 16600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:56.759140 16600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:56.760344 16600 hybrid_clock.cc:648] HybridClock initialized: now 1790606756760332 us; error 33 us; skew 500 ppm
Sep 28 14:45:56 dist-test-slave-gkhx krb5kdc[16587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606756, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:56.764086 16600 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:56.765100 16600 webserver.cc:460] Webserver started at http://127.13.223.126:43059/ using document root <none> and password file <none>
I20260928 14:45:56.765707 16600 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:56.765800 16600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:56.766050 16600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:56.768272 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "0974e6df1a114511801aadb3d40b0364"
format_stamp: "Formatted at 2026-09-28 14:45:56 on dist-test-slave-gkhx"
server_key: "3022077b84fd6b3e329f0238ff866895"
server_key_iv: "da6d437487eb2a39c9e952aebe6f46b3"
server_key_version: "encryptionkey@0"
I20260928 14:45:56.768873 16600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "0974e6df1a114511801aadb3d40b0364"
format_stamp: "Formatted at 2026-09-28 14:45:56 on dist-test-slave-gkhx"
server_key: "3022077b84fd6b3e329f0238ff866895"
server_key_iv: "da6d437487eb2a39c9e952aebe6f46b3"
server_key_version: "encryptionkey@0"
I20260928 14:45:56.773816 16600 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:45:56.776765 16615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:56.778223 16600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:56.778342 16600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "0974e6df1a114511801aadb3d40b0364"
format_stamp: "Formatted at 2026-09-28 14:45:56 on dist-test-slave-gkhx"
server_key: "3022077b84fd6b3e329f0238ff866895"
server_key_iv: "da6d437487eb2a39c9e952aebe6f46b3"
server_key_version: "encryptionkey@0"
I20260928 14:45:56.778447 16600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:56.789902 16600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:56.793416 16600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:56.793591 16600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:56.801143 16600 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34933
I20260928 14:45:56.801201 16667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34933 every 8 connection(s)
I20260928 14:45:56.802402 16600 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:56.805469 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16600
I20260928 14:45:56.805567 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:56.805861 14205 external_mini_cluster.cc:1428] Setting key 1a082d51aed7411418b52812d5ac42bf
I20260928 14:45:56.805873 16668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:56.813910 16668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364: Bootstrap starting.
I20260928 14:45:56.816422 16668 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364: Neither blocks nor log segments found. Creating new log.
Sep 28 14:45:56 dist-test-slave-gkhx krb5kdc[16587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:56.817351 16668 log.cc:824] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:56.820202 16668 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364: No bootstrap required, opened a new log
I20260928 14:45:56.823380 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:56.807346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.13.223.126:34933)
0928 14:45:56.807787 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:56.807807 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:56.807849 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:45:56.808962 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:56.808990 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:56.809492 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:56.809910 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:56.810810 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:56.812367 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:56.813455 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:56.814161 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:56.814320 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:56.817208 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:56.817218 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:56.817238 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:56.817276 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:56.819561 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:56.820173 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:56.820179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:56.820186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:56.820374 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:56.820722 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:56.820726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:56.820727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:56.821300 (+ 573us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:56.821505 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:56.821835 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 14:45:56.822172 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":115,"threads_started":1}
I20260928 14:45:56.825235 16668 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } }
I20260928 14:45:56.825502 16668 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:56.825572 16668 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0974e6df1a114511801aadb3d40b0364, State: Initialized, Role: FOLLOWER
I20260928 14:45:56.826085 16668 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } }
I20260928 14:45:56.826215 16668 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:56.826305 16668 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:56.826428 16668 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:56.827505 16668 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } }
I20260928 14:45:56.827895 16668 leader_election.cc:302] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0974e6df1a114511801aadb3d40b0364; no voters:
I20260928 14:45:56.828277 16668 leader_election.cc:288] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:56.828438 16673 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:56.828708 16673 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [term 1 LEADER]: Becoming Leader. State: Replica: 0974e6df1a114511801aadb3d40b0364, State: Running, Role: LEADER
I20260928 14:45:56.829089 16673 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } }
I20260928 14:45:56.829666 16668 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:56.830971 16674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0974e6df1a114511801aadb3d40b0364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } } }
I20260928 14:45:56.831084 16674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:56.831295 16675 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0974e6df1a114511801aadb3d40b0364. Latest consensus state: current_term: 1 leader_uuid: "0974e6df1a114511801aadb3d40b0364" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0974e6df1a114511801aadb3d40b0364" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34933 } } }
I20260928 14:45:56.831383 16675 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:56.831694 16682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:56.834839 16682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:56.841215 16682 catalog_manager.cc:1347] Generated new cluster ID: a3e1223bc2f14fffb00e7c8dbfa65265
I20260928 14:45:56.841310 16682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:56.876287 16682 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:56.877374 16682 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:56.884398 16682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0974e6df1a114511801aadb3d40b0364: Generated new TSK 0
I20260928 14:45:56.885247 16682 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:45:56.922432 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:57.048031 16694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:57.048352 16694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:57.048400 16694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:57.052922 16694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:57.053012 16694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:57.053076 16694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:45:57.059095 16694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:57.060340 16694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:57.061504 16694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:57.073314 16700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:57.073392 16694 server_base.cc:1034] running on GCE node
W20260928 14:45:57.073612 16702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:57.073304 16699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:57.074200 16694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:57.075044 16694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:57.076252 16694 hybrid_clock.cc:648] HybridClock initialized: now 1790606757076220 us; error 43 us; skew 500 ppm
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:57.079550 16694 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:45:57.080587 16694 webserver.cc:460] Webserver started at http://127.13.223.65:46783/ using document root <none> and password file <none>
I20260928 14:45:57.081171 16694 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:57.081233 16694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:57.081408 16694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:57.083827 16694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "a199a6cd6a9e4c69a32f96b5b7024e02"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "7b8a8853a8484fc5788e317f52b8f18b"
server_key_iv: "c5f957584944791cdf2aa935a8656a0a"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.084474 16694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "a199a6cd6a9e4c69a32f96b5b7024e02"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "7b8a8853a8484fc5788e317f52b8f18b"
server_key_iv: "c5f957584944791cdf2aa935a8656a0a"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.088786 16694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:45:57.092049 16709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.093654 16694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:57.093837 16694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "a199a6cd6a9e4c69a32f96b5b7024e02"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "7b8a8853a8484fc5788e317f52b8f18b"
server_key_iv: "c5f957584944791cdf2aa935a8656a0a"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.093990 16694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:57.172791 16694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:57.176789 16694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:57.177057 16694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:57.177757 16694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:57.178817 16694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:57.178892 16694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.178977 16694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:57.179039 16694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.189775 16694 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:33695
I20260928 14:45:57.189796 16822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:33695 every 8 connection(s)
I20260928 14:45:57.190865 16694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:45:57.191385 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16694
I20260928 14:45:57.191558 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:45:57.191852 14205 external_mini_cluster.cc:1428] Setting key 51a0a279826265ef52a41b557892dba1
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:45:57.209415 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.193048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:43397 (local address 127.13.223.126:34933)
0928 14:45:57.193208 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:57.193214 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:57.194803 (+ 1589us) server_negotiation.cc:407] Connection header received
0928 14:45:57.196225 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:57.196228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:57.196281 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:57.196360 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:57.198241 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.199081 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.201020 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.201305 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.201449 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.204844 (+ 3395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:57.204847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:57.204849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:57.204881 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:57.206837 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.207619 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.207623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.207625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.207747 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.208135 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.208139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.208141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.208360 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:57.208472 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:57.209083 (+ 611us) server_negotiation.cc:299] Negotiation successful
0928 14:45:57.209237 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:45:57.210358 16825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.193649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34933 (local address 127.13.223.65:43397)
0928 14:45:57.194640 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:57.194693 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:57.195864 (+ 1171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:57.196610 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:57.196625 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:57.197129 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:57.197931 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.197958 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.199281 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.199289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.200599 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.200611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.201525 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.201530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.201755 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.202661 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:57.202681 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:57.204447 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:57.207015 (+ 2568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.207024 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.207048 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.207498 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.207841 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.207844 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.207846 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.208023 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.208473 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:57.208483 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:57.208792 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 14:45:57.209016 (+ 224us) client_negotiation.cc:240] Negotiation successful
0928 14:45:57.209413 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":207,"threads_started":1}
I20260928 14:45:57.212011 16823 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34933
I20260928 14:45:57.212301 16823 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:57.212890 16823 heartbeater.cc:511] Master 127.13.223.126:34933 requested a full tablet report, sending...
I20260928 14:45:57.214973 16632 ts_manager.cc:194] Registered new tserver with Master: a199a6cd6a9e4c69a32f96b5b7024e02 (127.13.223.65:33695)
I20260928 14:45:57.216530 16632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:43397
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:45:57.227702 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:57.350311 16828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:57.350739 16828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:57.350816 16828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:57.355165 16828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:57.355237 16828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:57.355300 16828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:45:57.361188 16828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:57.362460 16828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:57.363617 16828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:57.374890 16836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:57.374907 16833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:57.375133 16828 server_base.cc:1034] running on GCE node
W20260928 14:45:57.374897 16834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:57.375818 16828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:57.376660 16828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:57.377905 16828 hybrid_clock.cc:648] HybridClock initialized: now 1790606757377864 us; error 76 us; skew 500 ppm
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:57.381340 16828 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:45:57.382445 16828 webserver.cc:460] Webserver started at http://127.13.223.66:34137/ using document root <none> and password file <none>
I20260928 14:45:57.383189 16828 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:57.383267 16828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:57.383447 16828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:57.386009 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "6b461b24cf6144afa9cb43ca2f82aa06"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "499d9ed5e7e81732b23f45805844c9a3"
server_key_iv: "9c8457f2d64aeaf05efbf6674a543530"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.386685 16828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "6b461b24cf6144afa9cb43ca2f82aa06"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "499d9ed5e7e81732b23f45805844c9a3"
server_key_iv: "9c8457f2d64aeaf05efbf6674a543530"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.391256 16828 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:45:57.394280 16843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.396238 16828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:57.396518 16828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "6b461b24cf6144afa9cb43ca2f82aa06"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "499d9ed5e7e81732b23f45805844c9a3"
server_key_iv: "9c8457f2d64aeaf05efbf6674a543530"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.396680 16828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:57.410539 16828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:57.415551 16828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:57.415808 16828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:57.416550 16828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:57.417548 16828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:57.417603 16828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.417649 16828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:57.417712 16828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.428021 16828 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35729
I20260928 14:45:57.428056 16956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35729 every 8 connection(s)
I20260928 14:45:57.429062 16828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:45:57.436218 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16828
I20260928 14:45:57.436371 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:45:57.436672 14205 external_mini_cluster.cc:1428] Setting key 63b7b4ffcdc23d1898156faa726ee389
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:45:57.446874 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.431416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:36311 (local address 127.13.223.126:34933)
0928 14:45:57.431578 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:57.431584 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:57.432978 (+ 1394us) server_negotiation.cc:407] Connection header received
0928 14:45:57.434164 (+ 1186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:57.434168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:57.434225 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:57.434329 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:57.436121 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.436844 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.438755 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.439003 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.439107 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.442481 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:57.442485 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:57.442488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:57.442519 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:57.444357 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.444995 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.444999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.445001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.445105 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.445501 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.445504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.445506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.445759 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:57.445880 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:57.446484 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 14:45:57.446687 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:45:57.447858 16959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.431902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34933 (local address 127.13.223.66:36311)
0928 14:45:57.432783 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:57.432833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:57.433841 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:57.434589 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:57.434598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:57.435141 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:57.435766 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.435785 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.437018 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.437022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.438584 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.438599 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.439213 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.439220 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.439512 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.440390 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:57.440416 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:57.442244 (+ 1828us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:57.444529 (+ 2285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.444536 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.444557 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.444869 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.445209 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.445212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.445215 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.445388 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.445881 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:57.445892 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:57.446201 (+ 309us) client_negotiation.cc:769] Sending connection context
0928 14:45:57.446420 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 14:45:57.446833 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":131,"threads_started":1}
I20260928 14:45:57.449576 16957 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:45:57.449915 16957 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:57.450531 16957 heartbeater.cc:511] Master 127.13.223.126:34933 requested a full tablet report, sending...
I20260928 14:45:57.451979 16632 ts_manager.cc:194] Registered new tserver with Master: 6b461b24cf6144afa9cb43ca2f82aa06 (127.13.223.66:35729)
I20260928 14:45:57.452637 16632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:36311
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:45:57.472162 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:57.597113 16962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:57.597431 16962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:57.597476 16962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:57.602108 16962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:57.602188 16962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:57.602257 16962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:45:57.608174 16962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34933
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:57.609449 16962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:57.610700 16962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:45:57.621421 16968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:57.621421 16970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:57.621524 16967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:57.621955 16962 server_base.cc:1034] running on GCE node
I20260928 14:45:57.622524 16962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:57.623339 16962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:57.624527 16962 hybrid_clock.cc:648] HybridClock initialized: now 1790606757624506 us; error 38 us; skew 500 ppm
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:57.627921 16962 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:45:57.628959 16962 webserver.cc:460] Webserver started at http://127.13.223.67:40639/ using document root <none> and password file <none>
I20260928 14:45:57.629621 16962 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:57.629750 16962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:57.629995 16962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:57.632652 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "a2fb3cd6b6ab4020af4cbce06874e1ec"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "b32b70c8236278d046aefd7bbedcd148"
server_key_iv: "34e563ef30848ed76c4cb6520d7d9a0c"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.633244 16962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "a2fb3cd6b6ab4020af4cbce06874e1ec"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "b32b70c8236278d046aefd7bbedcd148"
server_key_iv: "34e563ef30848ed76c4cb6520d7d9a0c"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.637728 16962 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:45:57.640851 16977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.642349 16962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:45:57.642529 16962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "a2fb3cd6b6ab4020af4cbce06874e1ec"
format_stamp: "Formatted at 2026-09-28 14:45:57 on dist-test-slave-gkhx"
server_key: "b32b70c8236278d046aefd7bbedcd148"
server_key_iv: "34e563ef30848ed76c4cb6520d7d9a0c"
server_key_version: "encryptionkey@0"
I20260928 14:45:57.642706 16962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:57.663779 16962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:57.667387 16962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:57.667623 16962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:57.668262 16962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:45:57.669291 16962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:45:57.669339 16962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.669407 16962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:45:57.669441 16962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:57.679600 16962 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:46483
I20260928 14:45:57.679615 17090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:46483 every 8 connection(s)
I20260928 14:45:57.680711 16962 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:45:57.691028 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 16962
I20260928 14:45:57.691192 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:45:57.691563 14205 external_mini_cluster.cc:1428] Setting key 99015ae2094852fa6c84d75194f6fb62
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[16587](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606757, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:57.697273 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.683096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41241 (local address 127.13.223.126:34933)
0928 14:45:57.683269 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:57.683273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:57.684514 (+ 1241us) server_negotiation.cc:407] Connection header received
0928 14:45:57.685534 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:57.685538 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:57.685606 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:57.685710 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:57.687098 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.687658 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.688953 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.689187 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.689306 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.692637 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:57.692641 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:57.692643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:57.692677 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:57.694765 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.695472 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.695475 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.695476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.695579 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:57.695953 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:57.695956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:57.695957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:57.696168 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:57.696246 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:57.696877 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 14:45:57.697112 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:45:57.698441 17093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:57.683538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34933 (local address 127.13.223.67:41241)
0928 14:45:57.684328 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0928 14:45:57.684373 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:45:57.685258 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:45:57.685914 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:45:57.685922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:45:57.686323 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:45:57.686916 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.686933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.687810 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.687816 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.688812 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:45:57.688823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:45:57.689372 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:45:57.689376 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:45:57.689573 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:57.690266 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:45:57.690284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:45:57.692363 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:45:57.694922 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.694930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.695013 (+ 83us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.695364 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.695708 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:45:57.695710 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:45:57.695712 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:45:57.695868 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:45:57.696293 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:45:57.696300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:45:57.696613 (+ 313us) client_negotiation.cc:769] Sending connection context
0928 14:45:57.696823 (+ 210us) client_negotiation.cc:240] Negotiation successful
0928 14:45:57.697219 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":138,"threads_started":1}
I20260928 14:45:57.700028 17091 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34933
I20260928 14:45:57.700335 17091 heartbeater.cc:464] Registering TS with master...
I20260928 14:45:57.700960 17091 heartbeater.cc:511] Master 127.13.223.126:34933 requested a full tablet report, sending...
I20260928 14:45:57.702354 16632 ts_manager.cc:194] Registered new tserver with Master: a2fb3cd6b6ab4020af4cbce06874e1ec (127.13.223.67:46483)
I20260928 14:45:57.703140 16632 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:41241
I20260928 14:45:57.707068 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:45:57.721819 16671 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:45:57.720182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42032 (local address 127.13.223.126:34933)
0928 14:45:57.720371 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:57.720374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:57.720395 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:45:57.720558 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:57.720561 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:57.720610 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:57.720684 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:57.721660 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42032: BlockingRecv error: recv got EOF from 127.0.0.1:42032 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:45:57.722929 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16694
I20260928 14:45:57.728629 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16828
I20260928 14:45:57.734885 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16962
I20260928 14:45:57.740871 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 16600
2026-09-28T14:45:57Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2386 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[17103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[17103](info): set up 2 sockets
Sep 28 14:45:57 dist-test-slave-gkhx krb5kdc[17103](info): commencing operation
krb5kdc: starting...
W20260928 14:45:58.837388 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:45:58 dist-test-slave-gkhx krb5kdc[17103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:45:58.983039 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46327
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44159
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:45:59.101778 17116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:45:59.102135 17116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:45:59.102211 17116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:45:59.106510 17116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:45:59.106673 17116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:45:59.106709 17116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:45:59.106751 17116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:45:59.106791 17116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:45:59.112437 17116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44159
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:45:59.113631 17116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:45:59.114781 17116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:45:59.124356 17116 server_base.cc:1034] running on GCE node
W20260928 14:45:59.124410 17124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:59.124341 17121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:45:59.124336 17122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:45:59.125260 17116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:45:59.126437 17116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:45:59.127601 17116 hybrid_clock.cc:648] HybridClock initialized: now 1790606759127585 us; error 38 us; skew 500 ppm
Sep 28 14:45:59 dist-test-slave-gkhx krb5kdc[17103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606759, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:45:59.130613 17116 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:45:59.131686 17116 webserver.cc:460] Webserver started at http://127.13.223.126:41921/ using document root <none> and password file <none>
I20260928 14:45:59.132293 17116 fs_manager.cc:359] Metadata directory not provided
I20260928 14:45:59.132393 17116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:45:59.132643 17116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:45:59.134948 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "19e472dfae6d466e84a1bdd30fde3560"
format_stamp: "Formatted at 2026-09-28 14:45:59 on dist-test-slave-gkhx"
server_key: "a55f6eb2e8fa6b35f31944e2c9e839a7"
server_key_iv: "20c57a3cfde03d595662fa47a860b448"
server_key_version: "encryptionkey@0"
I20260928 14:45:59.135592 17116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "19e472dfae6d466e84a1bdd30fde3560"
format_stamp: "Formatted at 2026-09-28 14:45:59 on dist-test-slave-gkhx"
server_key: "a55f6eb2e8fa6b35f31944e2c9e839a7"
server_key_iv: "20c57a3cfde03d595662fa47a860b448"
server_key_version: "encryptionkey@0"
I20260928 14:45:59.139961 17116 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:45:59.143275 17131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:45:59.144941 17116 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:45:59.145218 17116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "19e472dfae6d466e84a1bdd30fde3560"
format_stamp: "Formatted at 2026-09-28 14:45:59 on dist-test-slave-gkhx"
server_key: "a55f6eb2e8fa6b35f31944e2c9e839a7"
server_key_iv: "20c57a3cfde03d595662fa47a860b448"
server_key_version: "encryptionkey@0"
I20260928 14:45:59.145375 17116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:45:59.199327 17116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:45:59.203059 17116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:45:59.203296 17116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:45:59.210860 17116 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:46327
I20260928 14:45:59.211264 17183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:46327 every 8 connection(s)
I20260928 14:45:59.211975 17116 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:45:59.215142 17184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:45:59.221417 17184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: Bootstrap starting.
I20260928 14:45:59.221851 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17116
I20260928 14:45:59.221975 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:45:59.222280 14205 external_mini_cluster.cc:1428] Setting key 8f754498c2d0411fd9336ec8e3c2138d
I20260928 14:45:59.224519 17184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: Neither blocks nor log segments found. Creating new log.
I20260928 14:45:59.225312 17184 log.cc:824] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: Log is configured to *not* fsync() on all Append() calls
I20260928 14:45:59.227811 17184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: No bootstrap required, opened a new log
Sep 28 14:45:59 dist-test-slave-gkhx krb5kdc[17103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:45:59.232810 17184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } }
I20260928 14:45:59.233170 17184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:45:59.233228 17184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19e472dfae6d466e84a1bdd30fde3560, State: Initialized, Role: FOLLOWER
I20260928 14:45:59.233742 17184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } }
I20260928 14:45:59.233927 17184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:45:59.234022 17184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:45:59.234163 17184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:45:59.235217 17184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } }
I20260928 14:45:59.235608 17184 leader_election.cc:302] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19e472dfae6d466e84a1bdd30fde3560; no voters:
I20260928 14:45:59.235961 17184 leader_election.cc:288] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:45:59.236129 17189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:45:59.236426 17189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [term 1 LEADER]: Becoming Leader. State: Replica: 19e472dfae6d466e84a1bdd30fde3560, State: Running, Role: LEADER
I20260928 14:45:59.237434 17189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } }
I20260928 14:45:59.237576 17184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:45:59.239583 17191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19e472dfae6d466e84a1bdd30fde3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } } }
I20260928 14:45:59.239681 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:59.223761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.13.223.126:46327)
0928 14:45:59.224297 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:59.224313 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:59.224352 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:45:59.225104 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:59.225125 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:59.225546 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:59.225947 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:59.227131 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:59.228295 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:59.229531 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:59.230058 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:59.230207 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:59.233411 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:59.233417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:59.233460 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:45:59.233524 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:59.235881 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:59.236504 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:59.236511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:59.236519 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:59.236710 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:45:59.237050 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:45:59.237054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:45:59.237056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:45:59.237621 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:59.237830 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:59.238146 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 14:45:59.238523 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":175,"threads_started":1}
I20260928 14:45:59.239748 17191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [sys.catalog]: This master's current role is: LEADER
I20260928 14:45:59.240003 17190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19e472dfae6d466e84a1bdd30fde3560. Latest consensus state: current_term: 1 leader_uuid: "19e472dfae6d466e84a1bdd30fde3560" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19e472dfae6d466e84a1bdd30fde3560" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46327 } } }
I20260928 14:45:59.240085 17190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560 [sys.catalog]: This master's current role is: LEADER
W20260928 14:45:59.243503 17203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:45:59.243636 17203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:45:59.243680 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:45:59.243710 17204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:45:59.244545 17204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:45:59.251845 17204 catalog_manager.cc:1347] Generated new cluster ID: 1bcd8005ad954ecb998dd3868ddfc174
I20260928 14:45:59.251930 17204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:45:59.270764 17204 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:45:59.272141 17204 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:45:59.274633 17187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:45:59.268356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42772 (local address 127.13.223.126:46327)
0928 14:45:59.268496 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:45:59.268501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:45:59.268765 (+ 264us) server_negotiation.cc:407] Connection header received
0928 14:45:59.268942 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:45:59.268948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:45:59.269010 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:45:59.269089 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:45:59.269820 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:59.270552 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:59.271856 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:45:59.272226 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:45:59.272335 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:45:59.273504 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:45:59.273507 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:45:59.273510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:45:59.274081 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:45:59.274116 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:45:59.274134 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:45:59.274220 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:45:59.274365 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 14:45:59.274449 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:45:59.280382 17204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19e472dfae6d466e84a1bdd30fde3560: Generated new TSK 0
I20260928 14:45:59.281206 17204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:45:59.291841 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17116
2026-09-28T14:45:59Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1547 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:45:59 dist-test-slave-gkhx krb5kdc[17217](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:45:59 dist-test-slave-gkhx krb5kdc[17217](info): set up 2 sockets
Sep 28 14:45:59 dist-test-slave-gkhx krb5kdc[17217](info): commencing operation
krb5kdc: starting...
W20260928 14:46:00.397217 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260928 14:46:00.570044 14205 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.173s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260928 14:46:00.730770 14205 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.161s user 0.000s sys 0.002s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260928 14:46:00.962482 14205 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.232s user 0.000s sys 0.002s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/test-user.keytab.
W20260928 14:46:01.575148 14205 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.612s user 0.002s sys 0.001s
Sep 28 14:46:01 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
W20260928 14:46:02.540293 14205 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.223.126: real 0.937s user 0.002s sys 0.001s
I20260928 14:46:02.540894 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:33209
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:33209
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:02.691524 17230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:02.692296 17230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:02.692400 17230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:02.697192 17230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:02.697324 17230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:02.697361 17230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:02.697402 17230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:02.697440 17230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:02.703775 17230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:33209
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:33209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:02.705116 17230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:02.706315 17230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:02.716572 17235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:02.716768 17230 server_base.cc:1034] running on GCE node
W20260928 14:46:02.716579 17236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:02.716578 17238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:02.717720 17230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:02.719175 17230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:02.720436 17230 hybrid_clock.cc:648] HybridClock initialized: now 1790606762720414 us; error 47 us; skew 500 ppm
Sep 28 14:46:02 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606762, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:02.735623 17230 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:02.736967 17230 webserver.cc:460] Webserver started at http://127.13.223.126:33197/ using document root <none> and password file <none>
I20260928 14:46:02.737754 17230 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:02.737864 17230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:02.738138 17230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:02.740654 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "ba652a9dfa88428eb057e82503c593e3"
format_stamp: "Formatted at 2026-09-28 14:46:02 on dist-test-slave-gkhx"
server_key: "ad4a288b9cef8f1feeeed3d839cf7e45"
server_key_iv: "f3677b447e40f74fb17f7b519645470e"
server_key_version: "encryptionkey@0"
I20260928 14:46:02.741333 17230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "ba652a9dfa88428eb057e82503c593e3"
format_stamp: "Formatted at 2026-09-28 14:46:02 on dist-test-slave-gkhx"
server_key: "ad4a288b9cef8f1feeeed3d839cf7e45"
server_key_iv: "f3677b447e40f74fb17f7b519645470e"
server_key_version: "encryptionkey@0"
I20260928 14:46:02.745946 17230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:46:02.749430 17245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:02.751324 17230 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 14:46:02.751533 17230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "ba652a9dfa88428eb057e82503c593e3"
format_stamp: "Formatted at 2026-09-28 14:46:02 on dist-test-slave-gkhx"
server_key: "ad4a288b9cef8f1feeeed3d839cf7e45"
server_key_iv: "f3677b447e40f74fb17f7b519645470e"
server_key_version: "encryptionkey@0"
I20260928 14:46:02.751689 17230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:02.769166 17230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:02.773092 17230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:02.773341 17230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:02.782490 17230 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:33209
I20260928 14:46:02.783072 17297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:33209 every 8 connection(s)
I20260928 14:46:02.783896 17230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:02.787833 17298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:02.789999 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17230
I20260928 14:46:02.790271 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:02.790620 14205 external_mini_cluster.cc:1428] Setting key 876002a1b6c5a535c4c4f9f213e5546f
I20260928 14:46:02.796708 17298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: Bootstrap starting.
I20260928 14:46:02.800617 17298 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:02.801734 17298 log.cc:824] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:02.804791 17298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: No bootstrap required, opened a new log
Sep 28 14:46:02 dist-test-slave-gkhx krb5kdc[17217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606761, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:02.810385 17298 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } }
I20260928 14:46:02.810807 17298 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:02.810940 17298 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba652a9dfa88428eb057e82503c593e3, State: Initialized, Role: FOLLOWER
I20260928 14:46:02.811564 17298 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } }
I20260928 14:46:02.811800 17298 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:02.811895 17298 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:02.812036 17298 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:02.813035 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:02.792352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51038 (local address 127.13.223.126:33209)
0928 14:46:02.792992 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:02.793019 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:02.793074 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:46:02.794161 (+ 1087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:02.794194 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:02.794761 (+ 567us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:02.795275 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:02.796601 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:02.798890 (+ 2289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:02.800854 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:02.801718 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:02.802164 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:02.805713 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:02.805721 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:02.805741 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:02.805779 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:02.808715 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:02.809367 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:02.809386 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:02.809392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:02.809546 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:02.809933 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:02.809937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:02.809939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:02.811110 (+ 1171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:02.811433 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:02.811726 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:46:02.812111 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":152,"threads_started":1}
I20260928 14:46:02.890944 17298 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } }
I20260928 14:46:02.891599 17298 leader_election.cc:302] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba652a9dfa88428eb057e82503c593e3; no voters:
I20260928 14:46:02.892125 17298 leader_election.cc:288] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:02.892388 17303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:02.892684 17303 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [term 1 LEADER]: Becoming Leader. State: Replica: ba652a9dfa88428eb057e82503c593e3, State: Running, Role: LEADER
I20260928 14:46:02.893167 17303 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } }
I20260928 14:46:02.893814 17298 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:02.895586 17304 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba652a9dfa88428eb057e82503c593e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } } }
I20260928 14:46:02.895771 17304 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:02.896111 17305 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba652a9dfa88428eb057e82503c593e3. Latest consensus state: current_term: 1 leader_uuid: "ba652a9dfa88428eb057e82503c593e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba652a9dfa88428eb057e82503c593e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33209 } } }
I20260928 14:46:02.896348 17305 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3 [sys.catalog]: This master's current role is: LEADER
W20260928 14:46:02.899916 17318 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:46:02.900038 17318 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:46:02.900209 17315 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:02.901130 17315 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:02.908655 17315 catalog_manager.cc:1347] Generated new cluster ID: f2b6150c1e5f46f4bee631e4be5410a8
I20260928 14:46:02.908761 17315 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:02.929428 17315 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:02.930866 17315 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:02.939102 17315 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ba652a9dfa88428eb057e82503c593e3: Generated new TSK 0
I20260928 14:46:02.940012 17315 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
W20260928 14:46:03.754869 14205 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.13.223.65: real 0.799s user 0.004s sys 0.000s
I20260928 14:46:03.755431 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:03.883584 17324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:03.883916 17324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:03.883965 17324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:03.888387 17324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:03.888468 17324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:03.888535 17324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:03.894655 17324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:03.895961 17324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:03.897034 17324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:46:03.908027 17324 server_base.cc:1034] running on GCE node
W20260928 14:46:03.908142 17330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:03.908339 17332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:03.908138 17329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:03.908969 17324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:03.909759 17324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:03.910949 17324 hybrid_clock.cc:648] HybridClock initialized: now 1790606763910932 us; error 48 us; skew 500 ppm
Sep 28 14:46:03 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606763, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:03.914268 17324 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:03.915469 17324 webserver.cc:460] Webserver started at http://127.13.223.65:46057/ using document root <none> and password file <none>
I20260928 14:46:03.916124 17324 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:03.916222 17324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:03.916487 17324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:03.918905 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "7e630b1c172b439a8ad0f0964a34dd39"
format_stamp: "Formatted at 2026-09-28 14:46:03 on dist-test-slave-gkhx"
server_key: "468874e4db7cd647961a3da692fb490f"
server_key_iv: "594aed21d6b5a2af1db1432ddb1fd828"
server_key_version: "encryptionkey@0"
I20260928 14:46:03.919569 17324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "7e630b1c172b439a8ad0f0964a34dd39"
format_stamp: "Formatted at 2026-09-28 14:46:03 on dist-test-slave-gkhx"
server_key: "468874e4db7cd647961a3da692fb490f"
server_key_iv: "594aed21d6b5a2af1db1432ddb1fd828"
server_key_version: "encryptionkey@0"
I20260928 14:46:03.924288 17324 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:03.927727 17339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:03.929419 17324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:03.929625 17324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "7e630b1c172b439a8ad0f0964a34dd39"
format_stamp: "Formatted at 2026-09-28 14:46:03 on dist-test-slave-gkhx"
server_key: "468874e4db7cd647961a3da692fb490f"
server_key_iv: "594aed21d6b5a2af1db1432ddb1fd828"
server_key_version: "encryptionkey@0"
I20260928 14:46:03.929781 17324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:03.959017 17324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:03.963577 17324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:03.963865 17324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:03.964617 17324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:03.965608 17324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:03.965684 17324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:03.965761 17324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:03.965806 17324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:03.979710 17324 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:34783
I20260928 14:46:03.979755 17452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:34783 every 8 connection(s)
I20260928 14:46:03.980764 17324 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:03.984441 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17324
I20260928 14:46:03.984647 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:03.985011 14205 external_mini_cluster.cc:1428] Setting key 6ca25ecef156fc6dbc30178cb8d16325
Sep 28 14:46:03 dist-test-slave-gkhx krb5kdc[17217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606763, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:46:04.000744 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:03.983588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:51245 (local address 127.13.223.126:33209)
0928 14:46:03.983934 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:03.983940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:03.984867 (+ 927us) server_negotiation.cc:407] Connection header received
0928 14:46:03.986630 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:03.986633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:03.986693 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:03.986807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:03.989007 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:03.989983 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:03.991529 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:03.991984 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:03.992119 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:03.995763 (+ 3644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:03.995771 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:03.995777 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:03.995821 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:03.997689 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:03.998375 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:03.998379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:03.998381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:03.998524 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:03.999192 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:03.999196 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:03.999197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:03.999430 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:03.999592 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:04.000421 (+ 829us) server_negotiation.cc:299] Negotiation successful
0928 14:46:04.000587 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":100,"threads_started":1}
I20260928 14:46:04.002327 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:03.984078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33209 (local address 127.13.223.65:51245)
0928 14:46:03.984655 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:03.984720 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:03.986271 (+ 1551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:03.987192 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:03.987204 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:03.987869 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:03.988737 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:03.988769 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:03.990194 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:03.990197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:03.991352 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:03.991366 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:03.992181 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:03.992184 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:03.992371 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:03.993219 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:03.993241 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:03.995469 (+ 2228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:03.997847 (+ 2378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:03.997854 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:03.997872 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:03.998239 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:03.998723 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:03.998726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:03.998727 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:03.998930 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:03.999630 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:03.999637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:04.000057 (+ 420us) client_negotiation.cc:769] Sending connection context
0928 14:46:04.000339 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 14:46:04.000885 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":150,"threads_started":1}
I20260928 14:46:04.003993 17453 heartbeater.cc:347] Connected to a master server at 127.13.223.126:33209
I20260928 14:46:04.004328 17453 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:04.005055 17453 heartbeater.cc:511] Master 127.13.223.126:33209 requested a full tablet report, sending...
I20260928 14:46:04.007572 17262 ts_manager.cc:194] Registered new tserver with Master: 7e630b1c172b439a8ad0f0964a34dd39 (127.13.223.65:34783)
I20260928 14:46:04.008980 17262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:51245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:04.025138 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:04.151026 17459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:04.151356 17459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:04.151414 17459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:04.156181 17459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:04.156272 17459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:04.156342 17459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:04.162415 17459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:04.163671 17459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:04.164758 17459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:04.175586 17465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:04.175586 17464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:04.175590 17467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:04.175825 17459 server_base.cc:1034] running on GCE node
I20260928 14:46:04.176528 17459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:04.177423 17459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:04.178647 17459 hybrid_clock.cc:648] HybridClock initialized: now 1790606764178606 us; error 74 us; skew 500 ppm
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:04.181911 17459 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:04.183109 17459 webserver.cc:460] Webserver started at http://127.13.223.66:45629/ using document root <none> and password file <none>
I20260928 14:46:04.183689 17459 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:04.183753 17459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:04.183930 17459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:04.186090 17459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "4a6decd32a7e4b7b96472b98024d284a"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "8494a525a5f615743c97b924170a1f14"
server_key_iv: "0a3f34313aeab324cb6fff532609504a"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.186772 17459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "4a6decd32a7e4b7b96472b98024d284a"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "8494a525a5f615743c97b924170a1f14"
server_key_iv: "0a3f34313aeab324cb6fff532609504a"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.191267 17459 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:04.194337 17474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.196338 17459 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:04.196563 17459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "4a6decd32a7e4b7b96472b98024d284a"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "8494a525a5f615743c97b924170a1f14"
server_key_iv: "0a3f34313aeab324cb6fff532609504a"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.196717 17459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:04.212816 17459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:04.216801 17459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:04.217082 17459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:04.217801 17459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:04.218884 17459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:04.218964 17459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.219051 17459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:04.219099 17459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.229506 17459 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37591
I20260928 14:46:04.229532 17587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37591 every 8 connection(s)
I20260928 14:46:04.230646 17459 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:04.233287 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17459
I20260928 14:46:04.233465 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:04.233772 14205 external_mini_cluster.cc:1428] Setting key aebe8f0f8fdc3f5e16bd930e3d20353e
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:46:04.248557 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:04.232977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:48039 (local address 127.13.223.126:33209)
0928 14:46:04.233334 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:04.233341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:04.234440 (+ 1099us) server_negotiation.cc:407] Connection header received
0928 14:46:04.236145 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:04.236149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:04.236207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:04.236306 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:04.238038 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.238742 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.240181 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.240461 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.240688 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:04.243942 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:04.243948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:04.243951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:04.243985 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:04.245651 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.246280 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.246283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.246285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.246381 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.246846 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.246849 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.246850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.247205 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:04.247313 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:04.248149 (+ 836us) server_negotiation.cc:299] Negotiation successful
0928 14:46:04.248325 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208}
I20260928 14:46:04.249539 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:04.233576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33209 (local address 127.13.223.66:48039)
0928 14:46:04.234226 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:04.234291 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:04.235870 (+ 1579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:04.236536 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:04.236547 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:04.236962 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:04.237819 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:04.237844 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.238885 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.238888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:04.240016 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:04.240027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.240740 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.240745 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:04.240947 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:04.241695 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:04.241713 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:04.243706 (+ 1993us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:04.245782 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:04.245789 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:04.245806 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:04.246135 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:04.246481 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:04.246483 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:04.246485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:04.246745 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:04.247491 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:04.247497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:04.247815 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 14:46:04.248084 (+ 269us) client_negotiation.cc:240] Negotiation successful
0928 14:46:04.248501 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":169,"threads_started":1}
I20260928 14:46:04.251065 17588 heartbeater.cc:347] Connected to a master server at 127.13.223.126:33209
I20260928 14:46:04.251398 17588 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:04.251955 17588 heartbeater.cc:511] Master 127.13.223.126:33209 requested a full tablet report, sending...
I20260928 14:46:04.253232 17262 ts_manager.cc:194] Registered new tserver with Master: 4a6decd32a7e4b7b96472b98024d284a (127.13.223.66:37591)
I20260928 14:46:04.253975 17262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:48039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:04.271492 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:04.407812 17593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:04.408210 17593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:04.408293 17593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:04.412658 17593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:04.412793 17593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:04.412909 17593 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:04.418792 17593 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:33209
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:04.420078 17593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:04.421206 17593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:04.432273 17601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:04.432274 17598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:04.432273 17599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:04.432596 17593 server_base.cc:1034] running on GCE node
I20260928 14:46:04.433140 17593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:04.434059 17593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:04.435290 17593 hybrid_clock.cc:648] HybridClock initialized: now 1790606764435266 us; error 54 us; skew 500 ppm
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:04.439159 17593 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:04.440433 17593 webserver.cc:460] Webserver started at http://127.13.223.67:43599/ using document root <none> and password file <none>
I20260928 14:46:04.441092 17593 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:04.441166 17593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:04.441376 17593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:04.443939 17593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "4a92d1fb748549a585da253924dec939"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "c1e6363cf05c939ca39244fcc1cdb7cf"
server_key_iv: "4e3ce694055aeb6ebc04895a07fa50d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.444633 17593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "4a92d1fb748549a585da253924dec939"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "c1e6363cf05c939ca39244fcc1cdb7cf"
server_key_iv: "4e3ce694055aeb6ebc04895a07fa50d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.449680 17593 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:46:04.453118 17608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.455448 17593 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260928 14:46:04.455799 17593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "4a92d1fb748549a585da253924dec939"
format_stamp: "Formatted at 2026-09-28 14:46:04 on dist-test-slave-gkhx"
server_key: "c1e6363cf05c939ca39244fcc1cdb7cf"
server_key_iv: "4e3ce694055aeb6ebc04895a07fa50d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:04.456315 17593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:04.472337 17593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:04.477208 17593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:04.477488 17593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:04.478739 17593 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:04.479980 17593 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:04.480041 17593 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.480090 17593 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:04.480106 17593 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:04.491430 17593 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:41965
I20260928 14:46:04.491447 17721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:41965 every 8 connection(s)
I20260928 14:46:04.492564 17593 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:04.502015 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17593
I20260928 14:46:04.502139 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:04.502444 14205 external_mini_cluster.cc:1428] Setting key ebcc1c16da76b9b689b86ed6ebe79de5
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:04.513249 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:04.494905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:57387 (local address 127.13.223.126:33209)
0928 14:46:04.495081 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:04.495087 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:04.496558 (+ 1471us) server_negotiation.cc:407] Connection header received
0928 14:46:04.498460 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:04.498464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:04.498537 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:04.498905 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:04.500662 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.501315 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.503207 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.503485 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.503631 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:04.507477 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:04.507482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:04.507485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:04.507532 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:04.509812 (+ 2280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.510861 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.510865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.510866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.511016 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.511556 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.511571 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.511572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.511818 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:04.512134 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:04.512969 (+ 835us) server_negotiation.cc:299] Negotiation successful
0928 14:46:04.513110 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:46:04.514338 17724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:04.495505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33209 (local address 127.13.223.67:57387)
0928 14:46:04.496248 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:04.496431 (+ 183us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:04.497962 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:04.499238 (+ 1276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:04.499249 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:04.499735 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:04.500407 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:04.500427 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.501540 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.501544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:04.503033 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:04.503055 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.503689 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.503694 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:04.503928 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:04.504638 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:04.504661 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:04.507026 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:04.510125 (+ 3099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:04.510134 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:04.510154 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:04.510693 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:04.511168 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:04.511171 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:04.511172 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:04.511387 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:04.512170 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:04.512178 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:04.512600 (+ 422us) client_negotiation.cc:769] Sending connection context
0928 14:46:04.512809 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 14:46:04.513178 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":139,"threads_started":1}
I20260928 14:46:04.516788 17722 heartbeater.cc:347] Connected to a master server at 127.13.223.126:33209
I20260928 14:46:04.517148 17722 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:04.517783 17722 heartbeater.cc:511] Master 127.13.223.126:33209 requested a full tablet report, sending...
I20260928 14:46:04.519109 17262 ts_manager.cc:194] Registered new tserver with Master: 4a92d1fb748549a585da253924dec939 (127.13.223.67:41965)
I20260928 14:46:04.519874 17262 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:57387
I20260928 14:46:04.530524 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17217](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606764, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:04.561734 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:04.550640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.13.223.126:33209)
0928 14:46:04.550783 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:04.550787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:04.551179 (+ 392us) server_negotiation.cc:407] Connection header received
0928 14:46:04.551237 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:04.551240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:04.551288 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:04.551371 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:04.552325 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.552927 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.554089 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:04.554344 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:04.554449 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:04.557689 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:04.557693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:04.557695 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:04.557730 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:04.559246 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.559800 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.559803 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.559804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.559896 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:04.560334 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:04.560338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:04.560339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:04.560696 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:04.560877 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:04.561379 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 14:46:04.561542 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 14:46:04.562224 17262 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51048
I20260928 14:46:04.563759 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17324
I20260928 14:46:04.570971 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17459
I20260928 14:46:04.578272 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17593
I20260928 14:46:04.585153 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17230
2026-09-28T14:46:04Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (5297 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17734](info): set up 2 sockets
Sep 28 14:46:04 dist-test-slave-gkhx krb5kdc[17734](info): commencing operation
krb5kdc: starting...
W20260928 14:46:05.689416 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:05 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:05.830484 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41949
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:41949
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:05.951767 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:05.952111 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:05.952157 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:05.956591 17747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:05.956679 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:05.956700 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:05.956717 17747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:05.956733 17747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:05.963555 17747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:41949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:05.964831 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:05.966022 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:05.975976 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:05.976023 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:05.975976 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:05.976238 17747 server_base.cc:1034] running on GCE node
I20260928 14:46:05.977128 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:05.978655 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:05.979882 17747 hybrid_clock.cc:648] HybridClock initialized: now 1790606765979851 us; error 52 us; skew 500 ppm
Sep 28 14:46:05 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606765, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:05.983548 17747 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:05.984786 17747 webserver.cc:460] Webserver started at http://127.13.223.126:46265/ using document root <none> and password file <none>
I20260928 14:46:05.985419 17747 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:05.985488 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:05.985723 17747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:05.988101 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "1b830a83052e474295299dedc824b6a3"
format_stamp: "Formatted at 2026-09-28 14:46:05 on dist-test-slave-gkhx"
server_key: "1e21cee5f95404b6989539d6d9083f08"
server_key_iv: "b4299cf79d69623b5e2ccb8d7f5a53be"
server_key_version: "encryptionkey@0"
I20260928 14:46:05.988727 17747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "1b830a83052e474295299dedc824b6a3"
format_stamp: "Formatted at 2026-09-28 14:46:05 on dist-test-slave-gkhx"
server_key: "1e21cee5f95404b6989539d6d9083f08"
server_key_iv: "b4299cf79d69623b5e2ccb8d7f5a53be"
server_key_version: "encryptionkey@0"
I20260928 14:46:05.993636 17747 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:46:05.997051 17762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:05.998952 17747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:05.999118 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "1b830a83052e474295299dedc824b6a3"
format_stamp: "Formatted at 2026-09-28 14:46:05 on dist-test-slave-gkhx"
server_key: "1e21cee5f95404b6989539d6d9083f08"
server_key_iv: "b4299cf79d69623b5e2ccb8d7f5a53be"
server_key_version: "encryptionkey@0"
I20260928 14:46:05.999267 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:06.017107 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:06.020936 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:06.021224 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:06.029816 17747 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:41949
I20260928 14:46:06.030011 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:41949 every 8 connection(s)
I20260928 14:46:06.031064 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:06.034473 17815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:06.038836 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17747
I20260928 14:46:06.038985 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:06.039322 14205 external_mini_cluster.cc:1428] Setting key 340be4cfd37e2e9cb2bf13fcf3221522
I20260928 14:46:06.041795 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: Bootstrap starting.
I20260928 14:46:06.044353 17815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:06.045199 17815 log.cc:824] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:06.047501 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: No bootstrap required, opened a new log
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606765, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:06.052318 17815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } }
I20260928 14:46:06.052742 17815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:06.052829 17815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b830a83052e474295299dedc824b6a3, State: Initialized, Role: FOLLOWER
I20260928 14:46:06.053422 17815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } }
I20260928 14:46:06.053610 17815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:06.053694 17815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:06.053800 17815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:06.055197 17815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } }
I20260928 14:46:06.055667 17815 leader_election.cc:302] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b830a83052e474295299dedc824b6a3; no voters:
I20260928 14:46:06.056152 17815 leader_election.cc:288] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:06.056293 17820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:06.056651 17820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b830a83052e474295299dedc824b6a3, State: Running, Role: LEADER
I20260928 14:46:06.057071 17820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } }
I20260928 14:46:06.057711 17815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:06.057726 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.041213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.13.223.126:41949)
0928 14:46:06.041780 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.041798 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.041842 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:46:06.042818 (+ 976us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.042849 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.043271 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.043700 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.044713 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.045957 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.047129 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.047756 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.047965 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.051140 (+ 3175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.051149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.051173 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.051218 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.053496 (+ 2278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.054218 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.054225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.054231 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.054386 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.054847 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.054851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.054853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.055491 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.055716 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.056276 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.056642 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":150,"threads_started":1}
I20260928 14:46:06.058851 17821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b830a83052e474295299dedc824b6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } } }
I20260928 14:46:06.058984 17821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:06.059331 17822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b830a83052e474295299dedc824b6a3. Latest consensus state: current_term: 1 leader_uuid: "1b830a83052e474295299dedc824b6a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b830a83052e474295299dedc824b6a3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41949 } } }
I20260928 14:46:06.059428 17822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3 [sys.catalog]: This master's current role is: LEADER
W20260928 14:46:06.063742 17835 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:46:06.063850 17835 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:46:06.063925 17829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:06.064822 17829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:06.071940 17829 catalog_manager.cc:1347] Generated new cluster ID: c88222824a7e4ea79e246818cd39af95
I20260928 14:46:06.072109 17829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:06.095391 17829 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:06.096611 17829 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:06.102787 17829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b830a83052e474295299dedc824b6a3: Generated new TSK 0
I20260928 14:46:06.103790 17829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:46:06.142084 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:06.276291 17841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:06.276738 17841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:06.276818 17841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:06.282090 17841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:06.282187 17841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:06.282260 17841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:06.288708 17841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:06.289989 17841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:06.291168 17841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:06.301954 17847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.301941 17846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.301937 17849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:06.302224 17841 server_base.cc:1034] running on GCE node
I20260928 14:46:06.302902 17841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:06.303730 17841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:06.304929 17841 hybrid_clock.cc:648] HybridClock initialized: now 1790606766304895 us; error 46 us; skew 500 ppm
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:06.308565 17841 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:06.309711 17841 webserver.cc:460] Webserver started at http://127.13.223.65:37119/ using document root <none> and password file <none>
I20260928 14:46:06.310418 17841 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:06.310537 17841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:06.310848 17841 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:06.313330 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "c2362bae18ee4ff5b1d487d77482fe77"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "f130daa3c7457103924ba76ac9196160"
server_key_iv: "92d443246730e6982bc6f87e8c5edbdb"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.314033 17841 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "c2362bae18ee4ff5b1d487d77482fe77"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "f130daa3c7457103924ba76ac9196160"
server_key_iv: "92d443246730e6982bc6f87e8c5edbdb"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.318753 17841 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.002s
I20260928 14:46:06.322130 17856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.323887 17841 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:06.324057 17841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "c2362bae18ee4ff5b1d487d77482fe77"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "f130daa3c7457103924ba76ac9196160"
server_key_iv: "92d443246730e6982bc6f87e8c5edbdb"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.324210 17841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:06.350400 17841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:06.354029 17841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:06.354286 17841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:06.355055 17841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:06.356127 17841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:06.356181 17841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.356266 17841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:06.356299 17841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.367436 17841 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:35809
I20260928 14:46:06.367456 17969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:35809 every 8 connection(s)
I20260928 14:46:06.368528 17841 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:06.370834 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17841
I20260928 14:46:06.370993 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:06.371338 14205 external_mini_cluster.cc:1428] Setting key db1af089ed6f5b29b8618d40e3334b4a
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:46:06.388258 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.370837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:52891 (local address 127.13.223.126:41949)
0928 14:46:06.371124 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.371129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.372152 (+ 1023us) server_negotiation.cc:407] Connection header received
0928 14:46:06.374329 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.374332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.374388 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.374485 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.376262 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.377247 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.378813 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.379281 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.379445 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.383006 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.383010 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.383012 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.383046 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.384790 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.385735 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.385742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.385747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.385867 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.386635 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.386639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.386640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.386881 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.386978 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.387830 (+ 852us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.388019 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184}
I20260928 14:46:06.389104 17972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.371307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41949 (local address 127.13.223.65:52891)
0928 14:46:06.371955 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:06.372025 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:06.373999 (+ 1974us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:06.374782 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:06.374796 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:06.375438 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:06.376050 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.376069 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.377426 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.377430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.378667 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.378679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.379505 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.379509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.379727 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.380455 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:06.380474 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:06.382735 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:06.385060 (+ 2325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.385068 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.385086 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.385553 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.386003 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.386035 (+ 32us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.386040 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.386305 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.387008 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:06.387020 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:06.387454 (+ 434us) client_negotiation.cc:769] Sending connection context
0928 14:46:06.387725 (+ 271us) client_negotiation.cc:240] Negotiation successful
0928 14:46:06.388057 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":161,"threads_started":1}
I20260928 14:46:06.390760 17970 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41949
I20260928 14:46:06.391064 17970 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:06.391793 17970 heartbeater.cc:511] Master 127.13.223.126:41949 requested a full tablet report, sending...
I20260928 14:46:06.393822 17779 ts_manager.cc:194] Registered new tserver with Master: c2362bae18ee4ff5b1d487d77482fe77 (127.13.223.65:35809)
I20260928 14:46:06.395552 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:52891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:06.407776 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:06.525699 17975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:06.526111 17975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:06.526193 17975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:06.530869 17975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:06.530985 17975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:06.531085 17975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:06.536854 17975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:06.538112 17975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:06.539278 17975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:06.550678 17981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.550678 17980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.550717 17983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:06.551239 17975 server_base.cc:1034] running on GCE node
I20260928 14:46:06.551767 17975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:06.552570 17975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:06.553809 17975 hybrid_clock.cc:648] HybridClock initialized: now 1790606766553776 us; error 48 us; skew 500 ppm
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:06.557298 17975 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:06.558425 17975 webserver.cc:460] Webserver started at http://127.13.223.66:42887/ using document root <none> and password file <none>
I20260928 14:46:06.559192 17975 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:06.559273 17975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:06.559556 17975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:06.561972 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "97941e9c5c6b4c3c89bacb7600c74ede"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "117a7d08013f930870904e9743493e8f"
server_key_iv: "c08f2eebe97bbe710b45a7d467ac7bb5"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.562727 17975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "97941e9c5c6b4c3c89bacb7600c74ede"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "117a7d08013f930870904e9743493e8f"
server_key_iv: "c08f2eebe97bbe710b45a7d467ac7bb5"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.566942 17975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:06.569959 17990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.571764 17975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:06.572003 17975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "97941e9c5c6b4c3c89bacb7600c74ede"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "117a7d08013f930870904e9743493e8f"
server_key_iv: "c08f2eebe97bbe710b45a7d467ac7bb5"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.572171 17975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:06.607415 17975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:06.611935 17975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:06.612233 17975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:06.613168 17975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:06.614271 17975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:06.614354 17975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.614434 17975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:06.614480 17975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.625283 17975 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:43541
I20260928 14:46:06.625329 18103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:43541 every 8 connection(s)
I20260928 14:46:06.626283 17975 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:06.636353 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 17975
I20260928 14:46:06.636466 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:06.636785 14205 external_mini_cluster.cc:1428] Setting key 3b5057222b15b9225aba64bd696314a5
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:06.643182 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.628330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:40537 (local address 127.13.223.126:41949)
0928 14:46:06.628517 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.628522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.629584 (+ 1062us) server_negotiation.cc:407] Connection header received
0928 14:46:06.631130 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.631133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.631185 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.631264 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.632622 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.633184 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.634454 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.634792 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.634977 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.638010 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.638014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.638016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.638041 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.640318 (+ 2277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.641020 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.641026 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.641028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.641134 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.641601 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.641606 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.641608 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.641842 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.641966 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.642843 (+ 877us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.643048 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:46:06.644528 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.628823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41949 (local address 127.13.223.66:40537)
0928 14:46:06.629412 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:06.629468 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:06.630912 (+ 1444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:06.631477 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:06.631484 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:06.631869 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:06.632405 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.632419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.633354 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.633358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.634323 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.634334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.635051 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.635055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.635233 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.635910 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:06.635927 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:06.637783 (+ 1856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:06.640484 (+ 2701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.640491 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.640508 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.640848 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.641267 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.641272 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.641274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.641461 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.641966 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:06.641976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:06.642453 (+ 477us) client_negotiation.cc:769] Sending connection context
0928 14:46:06.642759 (+ 306us) client_negotiation.cc:240] Negotiation successful
0928 14:46:06.643197 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":150,"threads_started":1}
I20260928 14:46:06.646061 18104 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41949
I20260928 14:46:06.646370 18104 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:06.647010 18104 heartbeater.cc:511] Master 127.13.223.126:41949 requested a full tablet report, sending...
I20260928 14:46:06.648306 17779 ts_manager.cc:194] Registered new tserver with Master: 97941e9c5c6b4c3c89bacb7600c74ede (127.13.223.66:43541)
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:46:06.649035 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:40537
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:06.671494 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:06.796774 18109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:06.797138 18109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:06.797212 18109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:06.801685 18109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:06.801815 18109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:06.801926 18109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:06.807900 18109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41949
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:06.809172 18109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:06.810349 18109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:46:06.821723 18109 server_base.cc:1034] running on GCE node
W20260928 14:46:06.821698 18115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.821688 18117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:06.821962 18114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:06.822393 18109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:06.823181 18109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:06.824380 18109 hybrid_clock.cc:648] HybridClock initialized: now 1790606766824369 us; error 28 us; skew 500 ppm
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:06.827497 18109 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:06.828532 18109 webserver.cc:460] Webserver started at http://127.13.223.67:44415/ using document root <none> and password file <none>
I20260928 14:46:06.829200 18109 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:06.829265 18109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:06.829442 18109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:06.831651 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "830ccd9bf99b412ea469518642bd08c3"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "783a8375eb73034a22e3f5790e28f7ec"
server_key_iv: "d17ff04584a5365865c7a0b3074bf750"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.832224 18109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "830ccd9bf99b412ea469518642bd08c3"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "783a8375eb73034a22e3f5790e28f7ec"
server_key_iv: "d17ff04584a5365865c7a0b3074bf750"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.836426 18109 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:06.839478 18124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.841210 18109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:06.841408 18109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "830ccd9bf99b412ea469518642bd08c3"
format_stamp: "Formatted at 2026-09-28 14:46:06 on dist-test-slave-gkhx"
server_key: "783a8375eb73034a22e3f5790e28f7ec"
server_key_iv: "d17ff04584a5365865c7a0b3074bf750"
server_key_version: "encryptionkey@0"
I20260928 14:46:06.841570 18109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:06.873046 18109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:06.876752 18109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:06.876987 18109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:06.877682 18109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:06.878603 18109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:06.878677 18109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.878754 18109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:06.878798 18109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:06.888360 18109 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:39377
I20260928 14:46:06.888463 18237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:39377 every 8 connection(s)
I20260928 14:46:06.889392 18109 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:06.890311 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18109
I20260928 14:46:06.890444 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:06.890749 14205 external_mini_cluster.cc:1428] Setting key 5210a95fc159296008c9df532402ddc6
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:06.907682 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.892958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:40785 (local address 127.13.223.126:41949)
0928 14:46:06.893125 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.893134 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.894178 (+ 1044us) server_negotiation.cc:407] Connection header received
0928 14:46:06.895986 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.895990 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.896063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.896150 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.897540 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.898151 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.899667 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.899926 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.900056 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.903187 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.903193 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.903199 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.903239 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.905101 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.905919 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.905923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.905924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.906014 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.906432 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.906434 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.906435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.906729 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.906823 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.907430 (+ 607us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.907552 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:46:06.908595 18240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.893449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41949 (local address 127.13.223.67:40785)
0928 14:46:06.893988 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:06.894047 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:06.895741 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:06.896356 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:06.896364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:06.896796 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:06.897370 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.897386 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.898303 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.898307 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.899499 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:06.899511 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.900112 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.900117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:06.900337 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.901024 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:06.901045 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:06.902910 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:06.905257 (+ 2347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.905265 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.905287 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.905781 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.906166 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:06.906169 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:06.906170 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:06.906332 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:06.906848 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:06.906854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:06.907176 (+ 322us) client_negotiation.cc:769] Sending connection context
0928 14:46:06.907355 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 14:46:06.907699 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":133,"threads_started":1}
I20260928 14:46:06.909896 18238 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41949
I20260928 14:46:06.910153 18238 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:06.910725 18238 heartbeater.cc:511] Master 127.13.223.126:41949 requested a full tablet report, sending...
I20260928 14:46:06.911852 17779 ts_manager.cc:194] Registered new tserver with Master: 830ccd9bf99b412ea469518642bd08c3 (127.13.223.67:39377)
I20260928 14:46:06.912530 17779 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:40785
I20260928 14:46:06.917503 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:46:06.948068 18245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.935153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51266 (local address 127.13.223.65:35809)
0928 14:46:06.935549 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.935556 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.935579 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:46:06.935629 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.935635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.935825 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.935952 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.936806 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.937698 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.938808 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.939317 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.939421 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.943100 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.943115 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.943131 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.943166 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.945210 (+ 2044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.945815 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.945826 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.945828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.945954 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.946602 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.946608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.946613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.946984 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.947267 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.947594 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.947802 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":80,"threads_started":1}
Sep 28 14:46:06 dist-test-slave-gkhx krb5kdc[17734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:06.961861 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:06.952155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45682 (local address 127.13.223.126:41949)
0928 14:46:06.952324 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:06.952327 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:06.952343 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:06.952478 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:06.952480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:06.952537 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:06.952609 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:06.953348 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.953914 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.954846 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:06.955079 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:06.955168 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:06.958155 (+ 2987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:06.958158 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:06.958160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:06.958192 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:06.959875 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.960457 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.960461 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.960463 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.960559 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:06.960859 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:06.960863 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:06.960865 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:06.961132 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:06.961277 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:06.961542 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:46:06.961683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 14:46:06.962306 17779 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45682
I20260928 14:46:06.963052 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17841
I20260928 14:46:06.970053 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17975
I20260928 14:46:06.976095 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18109
I20260928 14:46:06.982062 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 17747
2026-09-28T14:46:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2395 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:07 dist-test-slave-gkhx krb5kdc[18250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:07 dist-test-slave-gkhx krb5kdc[18250](info): set up 2 sockets
Sep 28 14:46:07 dist-test-slave-gkhx krb5kdc[18250](info): commencing operation
krb5kdc: starting...
W20260928 14:46:08.085070 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:08.231792 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46193
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:08.365222 18263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:08.365557 18263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:08.365603 18263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:08.370154 18263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:08.370257 18263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:08.370286 18263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:08.370306 18263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:08.370321 18263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:08.376571 18263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:08.377805 18263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:08.378984 18263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:08.388446 18268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.388607 18263 server_base.cc:1034] running on GCE node
W20260928 14:46:08.388437 18271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:08.388554 18269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.389475 18263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:08.390805 18263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:08.392033 18263 hybrid_clock.cc:648] HybridClock initialized: now 1790606768391994 us; error 56 us; skew 500 ppm
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:08.403358 18263 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:08.404331 18263 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:08.404440 18263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:08.404692 18263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:08.407131 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "9c785faff03b43bdb890cefbb22f0f83"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "5a4ed2ed8a371feca9dd4a81c75e41e2"
server_key_iv: "b796408ddc0307a47196fa8c2706a2dc"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.407755 18263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "9c785faff03b43bdb890cefbb22f0f83"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "5a4ed2ed8a371feca9dd4a81c75e41e2"
server_key_iv: "b796408ddc0307a47196fa8c2706a2dc"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.412102 18263 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:08.415220 18277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:08.416749 18263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:08.416942 18263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "9c785faff03b43bdb890cefbb22f0f83"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "5a4ed2ed8a371feca9dd4a81c75e41e2"
server_key_iv: "b796408ddc0307a47196fa8c2706a2dc"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.417114 18263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:08.444345 18263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:08.448374 18263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:08.448619 18263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:08.457006 18263 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:46193
I20260928 14:46:08.457399 18329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:46193 every 8 connection(s)
I20260928 14:46:08.457862 18263 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:08.460721 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18263
I20260928 14:46:08.460896 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:08.461236 14205 external_mini_cluster.cc:1428] Setting key 7064f8c7a01d35c683f760abed746bc8
I20260928 14:46:08.461591 18330 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:08.470382 18330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: Bootstrap starting.
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:08.473532 18330 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:08.474628 18330 log.cc:824] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:08.477751 18330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: No bootstrap required, opened a new log
I20260928 14:46:08.479072 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:08.462905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.13.223.126:46193)
0928 14:46:08.463381 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:08.463400 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:08.463439 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:46:08.464133 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:08.464152 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:08.464529 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:08.464847 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:08.466154 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.467489 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.469032 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.469627 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.469774 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:08.473079 (+ 3305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:08.473086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:08.473102 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:08.473141 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:08.475286 (+ 2145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:08.475990 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:08.475998 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:08.476004 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:08.476156 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:08.476525 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:08.476528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:08.476529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:08.476996 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:08.477175 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:08.477592 (+ 417us) server_negotiation.cc:299] Negotiation successful
0928 14:46:08.478151 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":164,"threads_started":1}
I20260928 14:46:08.482075 18330 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } }
I20260928 14:46:08.482431 18330 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:08.482609 18330 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c785faff03b43bdb890cefbb22f0f83, State: Initialized, Role: FOLLOWER
I20260928 14:46:08.483119 18330 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } }
I20260928 14:46:08.483278 18330 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:08.483372 18330 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:08.483502 18330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:08.484805 18330 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } }
I20260928 14:46:08.485227 18330 leader_election.cc:302] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c785faff03b43bdb890cefbb22f0f83; no voters:
I20260928 14:46:08.485601 18330 leader_election.cc:288] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:08.485750 18335 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:08.486065 18335 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [term 1 LEADER]: Becoming Leader. State: Replica: 9c785faff03b43bdb890cefbb22f0f83, State: Running, Role: LEADER
I20260928 14:46:08.486519 18335 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } }
I20260928 14:46:08.487164 18330 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:08.489079 18337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c785faff03b43bdb890cefbb22f0f83. Latest consensus state: current_term: 1 leader_uuid: "9c785faff03b43bdb890cefbb22f0f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } } }
I20260928 14:46:08.489193 18337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:08.489006 18336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c785faff03b43bdb890cefbb22f0f83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c785faff03b43bdb890cefbb22f0f83" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46193 } } }
I20260928 14:46:08.489565 18336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:08.490293 18347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:08.494050 18347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:08.501359 18347 catalog_manager.cc:1347] Generated new cluster ID: 814b3f963c304b17b87d1b799154960f
I20260928 14:46:08.501470 18347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:08.520747 18347 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:08.522194 18347 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:08.539582 18347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: Generated new TSK 0
I20260928 14:46:08.540937 18347 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:46:08.591317 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:08.721117 18356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:08.721510 18356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:08.721582 18356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:08.726243 18356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:08.726341 18356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:08.726408 18356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:08.732723 18356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:08.734002 18356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:08.735208 18356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:08.746582 18362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:08.746616 18361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.746790 18356 server_base.cc:1034] running on GCE node
W20260928 14:46:08.746610 18364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.747551 18356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:08.748364 18356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:08.749598 18356 hybrid_clock.cc:648] HybridClock initialized: now 1790606768749583 us; error 33 us; skew 500 ppm
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:08.753044 18356 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:08.753895 18356 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:08.754026 18356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:08.754278 18356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:08.756583 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "e10fcbe712464d4387591a8a8039c541"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "b58ad404698919db5c17e50d6163f5a7"
server_key_iv: "d48d7c65bf7b3bf89a564f0b1094953c"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.757200 18356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "e10fcbe712464d4387591a8a8039c541"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "b58ad404698919db5c17e50d6163f5a7"
server_key_iv: "d48d7c65bf7b3bf89a564f0b1094953c"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.761435 18356 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:08.764413 18370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:08.766290 18356 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:08.766484 18356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "e10fcbe712464d4387591a8a8039c541"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "b58ad404698919db5c17e50d6163f5a7"
server_key_iv: "d48d7c65bf7b3bf89a564f0b1094953c"
server_key_version: "encryptionkey@0"
I20260928 14:46:08.766726 18356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:08.781188 18356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:08.785504 18356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:08.785765 18356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:08.786497 18356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:08.787755 18356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:08.787860 18356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:08.787951 18356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:08.787984 18356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:08.798661 18356 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:37363
I20260928 14:46:08.798695 18483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:37363 every 8 connection(s)
I20260928 14:46:08.799561 18356 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:08.809764 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18356
I20260928 14:46:08.809898 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:08.810367 14205 external_mini_cluster.cc:1428] Setting key 9fa0fe2e43a333f1763dcf274b49df8d
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:08.818043 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:08.801789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:33405 (local address 127.13.223.126:46193)
0928 14:46:08.801972 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:08.801976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:08.803643 (+ 1667us) server_negotiation.cc:407] Connection header received
0928 14:46:08.804950 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:08.804954 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:08.805028 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:08.805158 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:08.806939 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.807560 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.808979 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.809260 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.809420 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:08.813142 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:08.813150 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:08.813156 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:08.813194 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:08.814987 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:08.815819 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:08.815825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:08.815830 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:08.815948 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:08.816371 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:08.816378 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:08.816380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:08.816753 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:08.816879 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:08.817630 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 14:46:08.817812 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:46:08.818718 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:08.802351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46193 (local address 127.13.223.65:33405)
0928 14:46:08.803399 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:08.803452 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:08.804521 (+ 1069us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:08.805422 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:08.805447 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:08.806020 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:08.806731 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:08.806751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.807739 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.807742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:08.808805 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:08.808817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:08.809499 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:08.809504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:08.809713 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:08.810754 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:08.810776 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:08.812930 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:08.815138 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:08.815149 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:08.815173 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:08.815632 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:08.816074 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:08.816076 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:08.816078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:08.816244 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:08.816871 (+ 627us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:08.816876 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:08.817161 (+ 285us) client_negotiation.cc:769] Sending connection context
0928 14:46:08.817348 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 14:46:08.817660 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":203,"threads_started":1}
I20260928 14:46:08.820209 18484 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46193
I20260928 14:46:08.820490 18484 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:08.821002 18484 heartbeater.cc:511] Master 127.13.223.126:46193 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:46:08.822883 18294 ts_manager.cc:194] Registered new tserver with Master: e10fcbe712464d4387591a8a8039c541 (127.13.223.65:37363)
I20260928 14:46:08.824666 18294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:33405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:08.846765 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:08.967808 18489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:08.968181 18489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:08.968277 18489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:08.973152 18489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:08.973258 18489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:08.973359 18489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:08.979358 18489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:08.980631 18489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:08.981768 18489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:08.992851 18497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.993341 18489 server_base.cc:1034] running on GCE node
W20260928 14:46:08.992851 18495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:08.992851 18494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:08.994139 18489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:08.995025 18489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:08.996259 18489 hybrid_clock.cc:648] HybridClock initialized: now 1790606768996214 us; error 62 us; skew 500 ppm
Sep 28 14:46:08 dist-test-slave-gkhx krb5kdc[18250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:08.999788 18489 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:09.000689 18489 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:09.000800 18489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:09.001070 18489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:09.003469 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "6a56be9d05a14dc782220ddfe5cd31c5"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "63d01d75b0502879d1945c32ddb82c17"
server_key_iv: "0016c119e0e18c3f4e18171a746d1f31"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.004199 18489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "6a56be9d05a14dc782220ddfe5cd31c5"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "63d01d75b0502879d1945c32ddb82c17"
server_key_iv: "0016c119e0e18c3f4e18171a746d1f31"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.008687 18489 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:09.011862 18503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.013362 18489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:09.013573 18489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "6a56be9d05a14dc782220ddfe5cd31c5"
format_stamp: "Formatted at 2026-09-28 14:46:08 on dist-test-slave-gkhx"
server_key: "63d01d75b0502879d1945c32ddb82c17"
server_key_iv: "0016c119e0e18c3f4e18171a746d1f31"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.013727 18489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:09.024479 18489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:09.029592 18489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:09.029812 18489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:09.030627 18489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:09.031729 18489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:09.031798 18489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.031888 18489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:09.031917 18489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.044229 18489 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35145
I20260928 14:46:09.044359 18616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35145 every 8 connection(s)
I20260928 14:46:09.044996 18489 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:09.054951 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18489
I20260928 14:46:09.055138 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:09.055491 14205 external_mini_cluster.cc:1428] Setting key 49fa375f9a7a0253fbbe7618f792063d
Sep 28 14:46:09 dist-test-slave-gkhx krb5kdc[18250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606768, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:09.064131 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.047577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:43695 (local address 127.13.223.126:46193)
0928 14:46:09.047765 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.047772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.049426 (+ 1654us) server_negotiation.cc:407] Connection header received
0928 14:46:09.050390 (+ 964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.050394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.050461 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.050721 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:09.052150 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.052813 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.054320 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.054684 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.054832 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.058663 (+ 3831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:09.058667 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:09.058670 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:09.058703 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:09.060934 (+ 2231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.061689 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.061694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.061696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.061807 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.062376 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.062384 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.062386 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.062779 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:09.062903 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.063622 (+ 719us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.063830 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:46:09.064833 18619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.048143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46193 (local address 127.13.223.66:43695)
0928 14:46:09.049172 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.049239 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.050119 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.050904 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.050913 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.051353 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:09.051943 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.051961 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.053062 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.053066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.054110 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.054123 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.054911 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.054915 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.055217 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.056073 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:09.056094 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:09.058381 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:09.061153 (+ 2772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:09.061160 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:09.061178 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:09.061552 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:09.061935 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:09.061938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:09.061939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:09.062114 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:09.062953 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:09.062960 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:09.063312 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.063500 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.063817 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":151,"threads_started":1}
I20260928 14:46:09.066543 18617 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46193
I20260928 14:46:09.066864 18617 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:46:09.067497 18617 heartbeater.cc:511] Master 127.13.223.126:46193 requested a full tablet report, sending...
I20260928 14:46:09.068794 18294 ts_manager.cc:194] Registered new tserver with Master: 6a56be9d05a14dc782220ddfe5cd31c5 (127.13.223.66:35145)
I20260928 14:46:09.069669 18294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:43695
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:09.090849 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:34737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:09.212883 18622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:09.213200 18622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:09.213250 18622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:09.217643 18622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:09.217712 18622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:09.217779 18622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:09.223686 18622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:46193
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:09.224920 18622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:09.225944 18622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:09.237054 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:09.237167 18622 server_base.cc:1034] running on GCE node
W20260928 14:46:09.237177 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:09.237073 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:09.237990 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:09.238802 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:09.240000 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790606769239977 us; error 41 us; skew 500 ppm
Sep 28 14:46:09 dist-test-slave-gkhx krb5kdc[18250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606769, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:09.244093 18622 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:09.245069 18622 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:09.245180 18622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:09.245416 18622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:09.247716 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "f235831c56f2496cadbc2415c3d6b44b"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5efd7d039fb6275d5d288c48a858f93f"
server_key_iv: "228108cbfd3d45c74f56cbf331499bc8"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.248346 18622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "f235831c56f2496cadbc2415c3d6b44b"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5efd7d039fb6275d5d288c48a858f93f"
server_key_iv: "228108cbfd3d45c74f56cbf331499bc8"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.252580 18622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:09.255620 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.257241 18622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:09.257452 18622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "f235831c56f2496cadbc2415c3d6b44b"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5efd7d039fb6275d5d288c48a858f93f"
server_key_iv: "228108cbfd3d45c74f56cbf331499bc8"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.257602 18622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:09.293426 18622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:09.297690 18622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:09.297955 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:09.298729 18622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:09.299655 18622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:09.299728 18622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.299795 18622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:09.299839 18622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.309988 18622 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:42591
I20260928 14:46:09.310046 18749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:42591 every 8 connection(s)
I20260928 14:46:09.310773 18622 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:09.318926 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18622
I20260928 14:46:09.319169 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:09.319468 14205 external_mini_cluster.cc:1428] Setting key 74d75729b59c0d777702a6628272d315
Sep 28 14:46:09 dist-test-slave-gkhx krb5kdc[18250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606769, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:09.327936 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.313029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:50707 (local address 127.13.223.126:46193)
0928 14:46:09.313189 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.313193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.314573 (+ 1380us) server_negotiation.cc:407] Connection header received
0928 14:46:09.315671 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.315675 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.315740 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.315861 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:09.317281 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.317999 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.319704 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.320065 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.320220 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.323712 (+ 3492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:09.323716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:09.323719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:09.323750 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:09.325483 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.326131 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.326135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.326136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.326224 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.326675 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.326678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.326679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.326911 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:09.327025 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.327612 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.327756 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:09.328944 18752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.313377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46193 (local address 127.13.223.67:50707)
0928 14:46:09.314328 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.314378 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.315418 (+ 1040us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.316069 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.316078 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.316510 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:09.317101 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.317119 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.318183 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.318187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.319540 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.319553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.320343 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.320352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.320635 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.321553 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:09.321573 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:09.323463 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:09.325654 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:09.325661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:09.325687 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:09.325996 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:09.326329 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:09.326332 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:09.326334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:09.326512 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:09.327039 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:09.327049 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:09.327345 (+ 296us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.327557 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.327952 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":135,"threads_started":1}
I20260928 14:46:09.330374 18750 heartbeater.cc:347] Connected to a master server at 127.13.223.126:46193
I20260928 14:46:09.330716 18750 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:09.331329 18750 heartbeater.cc:511] Master 127.13.223.126:46193 requested a full tablet report, sending...
I20260928 14:46:09.332647 18294 ts_manager.cc:194] Registered new tserver with Master: f235831c56f2496cadbc2415c3d6b44b (127.13.223.67:42591)
I20260928 14:46:09.333722 18294 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:50707
I20260928 14:46:09.334806 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:46:09.346683 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.337279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.13.223.126:46193)
0928 14:46:09.337445 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.337449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.337519 (+ 70us) server_negotiation.cc:407] Connection header received
0928 14:46:09.337638 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.337640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.337686 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.337765 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:09.338744 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.339405 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.340478 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.340707 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.340801 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.342754 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:09.342758 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:09.342760 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:09.342794 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:09.344639 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.345203 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.345206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.345208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.345305 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:09.345587 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:09.345590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:09.345591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:09.345859 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:09.345988 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.346254 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.346417 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:46:09.350662 18294 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:09.354175 18294 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:09.373427 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.367135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59710 (local address 127.13.223.65:37363)
0928 14:46:09.367915 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.367924 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.367949 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:46:09.368030 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.368035 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.368268 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.368850 (+ 582us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:09.368968 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.369879 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.371557 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.372650 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.372782 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.372965 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.373149 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.373258 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":106,"threads_started":1}
I20260928 14:46:09.373418 18762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.366961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:37363 (local address 127.0.0.1:59710)
0928 14:46:09.367366 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.367433 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.367580 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.368460 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.368467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.368492 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:09.368674 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.368682 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.370005 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.370008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.371444 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.371454 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.372781 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.372784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.372914 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.372955 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.373102 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.373254 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":82,"threads_started":1}
I20260928 14:46:09.373768 18766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.368303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35145 (local address 127.0.0.1:54822)
0928 14:46:09.368687 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.368700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.368773 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.369524 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.369527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.369546 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:09.369629 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.369635 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.371063 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.371070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.372269 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.372280 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.373328 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.373335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.373459 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.373482 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.373533 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.373608 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":2,"thread_start_us":95,"threads_started":1}
I20260928 14:46:09.374146 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.367969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46834 (local address 127.13.223.67:42591)
0928 14:46:09.368507 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.368520 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.368548 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:46:09.368726 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.368735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.368961 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.369093 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:09.369565 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.370812 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.372166 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.373037 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.373150 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.373318 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.373865 (+ 547us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.373986 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":188,"threads_started":1}
I20260928 14:46:09.374429 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.368479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54822 (local address 127.13.223.66:35145)
0928 14:46:09.369007 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.369025 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.369050 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:46:09.369119 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.369124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.369306 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.369531 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:09.369735 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.370867 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.372380 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.373165 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.373957 (+ 792us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.374109 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.374160 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.374261 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":100,"threads_started":1}
I20260928 14:46:09.374590 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.367298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42591 (local address 127.0.0.1:46834)
0928 14:46:09.368053 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.368069 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.368554 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.369287 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.369289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.369306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:09.369432 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.369440 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.370936 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.370940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.372029 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.372038 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.373371 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.373374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.373688 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.373709 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.374326 (+ 617us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.374406 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"mutex_wait_us":213,"thread_start_us":133,"threads_started":1}
I20260928 14:46:09.377261 18551 tablet_service.cc:1467] Processing CreateTablet for tablet 8996d2b9c51f4484abf70b8186740683 (DEFAULT_TABLE table=test-table [id=381572659e21439aa106e8125606202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:09.378505 18551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8996d2b9c51f4484abf70b8186740683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:09.380153 18684 tablet_service.cc:1467] Processing CreateTablet for tablet 8996d2b9c51f4484abf70b8186740683 (DEFAULT_TABLE table=test-table [id=381572659e21439aa106e8125606202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:09.380182 18418 tablet_service.cc:1467] Processing CreateTablet for tablet 8996d2b9c51f4484abf70b8186740683 (DEFAULT_TABLE table=test-table [id=381572659e21439aa106e8125606202c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:09.381208 18684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8996d2b9c51f4484abf70b8186740683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:09.381235 18418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8996d2b9c51f4484abf70b8186740683. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:09.386876 18768 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Bootstrap starting.
I20260928 14:46:09.388482 18769 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Bootstrap starting.
I20260928 14:46:09.389549 18770 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: Bootstrap starting.
I20260928 14:46:09.390502 18768 tablet_bootstrap.cc:654] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:09.391161 18769 tablet_bootstrap.cc:654] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:09.391952 18768 log.cc:824] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:09.392163 18770 tablet_bootstrap.cc:654] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:09.392168 18769 log.cc:824] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:09.393036 18770 log.cc:824] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:09.394448 18768 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: No bootstrap required, opened a new log
I20260928 14:46:09.394452 18769 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: No bootstrap required, opened a new log
I20260928 14:46:09.394718 18769 ts_tablet_manager.cc:1397] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:46:09.394718 18768 ts_tablet_manager.cc:1397] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 14:46:09.395002 18770 tablet_bootstrap.cc:492] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: No bootstrap required, opened a new log
I20260928 14:46:09.395213 18770 ts_tablet_manager.cc:1397] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:46:09.399194 18770 raft_consensus.cc:348] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.399104 18768 raft_consensus.cc:348] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.399104 18769 raft_consensus.cc:348] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.399473 18769 raft_consensus.cc:374] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:09.399473 18768 raft_consensus.cc:374] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:09.399473 18770 raft_consensus.cc:374] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:09.399544 18770 raft_consensus.cc:729] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e10fcbe712464d4387591a8a8039c541, State: Initialized, Role: FOLLOWER
I20260928 14:46:09.399555 18768 raft_consensus.cc:729] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a56be9d05a14dc782220ddfe5cd31c5, State: Initialized, Role: FOLLOWER
I20260928 14:46:09.399586 18769 raft_consensus.cc:729] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f235831c56f2496cadbc2415c3d6b44b, State: Initialized, Role: FOLLOWER
I20260928 14:46:09.400076 18770 consensus_queue.cc:260] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.400084 18768 consensus_queue.cc:260] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.400084 18769 consensus_queue.cc:260] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.401329 18768 ts_tablet_manager.cc:1428] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:09.401325 18750 heartbeater.cc:503] Master 127.13.223.126:46193 was elected leader, sending a full tablet report...
I20260928 14:46:09.401376 18770 ts_tablet_manager.cc:1428] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:46:09.401336 18769 ts_tablet_manager.cc:1428] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:09.401775 18617 heartbeater.cc:503] Master 127.13.223.126:46193 was elected leader, sending a full tablet report...
I20260928 14:46:09.401801 18484 heartbeater.cc:503] Master 127.13.223.126:46193 was elected leader, sending a full tablet report...
I20260928 14:46:09.476713 18776 raft_consensus.cc:482] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:09.476986 18776 raft_consensus.cc:504] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.478425 18776 leader_election.cc:288] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f235831c56f2496cadbc2415c3d6b44b (127.13.223.67:42591), 6a56be9d05a14dc782220ddfe5cd31c5 (127.13.223.66:35145)
I20260928 14:46:09.483633 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.478867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:44705 (local address 127.13.223.67:42591)
0928 14:46:09.479100 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.479104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.479292 (+ 188us) server_negotiation.cc:407] Connection header received
0928 14:46:09.479376 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.479383 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.479448 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.479599 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:09.480066 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.480819 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.482194 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.482900 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.483021 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.483069 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.483449 (+ 380us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.483527 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260928 14:46:09.483644 18777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.478765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:42591 (local address 127.13.223.65:44705)
0928 14:46:09.479121 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.479139 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.479265 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.479638 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.479642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.479665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:09.479842 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.479850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.480970 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.480974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.482006 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.482017 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.483089 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.483093 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.483203 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.483233 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.483339 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.483435 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":1,"thread_start_us":151,"threads_started":1}
I20260928 14:46:09.484521 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.479488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:40535 (local address 127.13.223.66:35145)
0928 14:46:09.479638 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.479642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.479680 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:46:09.479866 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.479871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.479930 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.480008 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:09.480454 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.481022 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.483072 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.483728 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.483858 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.483913 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.484306 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.484398 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:46:09.484542 18778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.479246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35145 (local address 127.13.223.65:40535)
0928 14:46:09.479602 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.479617 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:09.479740 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:09.480072 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:09.480076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:09.480100 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:09.480238 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.480246 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.481176 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.481179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.482783 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:09.482796 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.484045 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.484049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:09.484155 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.484176 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:46:09.484285 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:46:09.484369 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"spinlock_wait_cycles":4480,"thread_start_us":92,"threads_started":1}
I20260928 14:46:09.484661 18704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8996d2b9c51f4484abf70b8186740683" candidate_uuid: "e10fcbe712464d4387591a8a8039c541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f235831c56f2496cadbc2415c3d6b44b" is_pre_election: true
I20260928 14:46:09.485085 18704 raft_consensus.cc:2454] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e10fcbe712464d4387591a8a8039c541 in term 0.
I20260928 14:46:09.485364 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8996d2b9c51f4484abf70b8186740683" candidate_uuid: "e10fcbe712464d4387591a8a8039c541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" is_pre_election: true
I20260928 14:46:09.485695 18571 raft_consensus.cc:2454] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e10fcbe712464d4387591a8a8039c541 in term 0.
I20260928 14:46:09.485706 18372 leader_election.cc:302] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e10fcbe712464d4387591a8a8039c541, f235831c56f2496cadbc2415c3d6b44b; no voters:
I20260928 14:46:09.486119 18776 raft_consensus.cc:2781] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:09.486260 18776 raft_consensus.cc:482] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:09.486325 18776 raft_consensus.cc:3037] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:09.487633 18776 raft_consensus.cc:504] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.488057 18776 leader_election.cc:288] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [CANDIDATE]: Term 1 election: Requested vote from peers f235831c56f2496cadbc2415c3d6b44b (127.13.223.67:42591), 6a56be9d05a14dc782220ddfe5cd31c5 (127.13.223.66:35145)
I20260928 14:46:09.488617 18704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8996d2b9c51f4484abf70b8186740683" candidate_uuid: "e10fcbe712464d4387591a8a8039c541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f235831c56f2496cadbc2415c3d6b44b"
I20260928 14:46:09.488700 18571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8996d2b9c51f4484abf70b8186740683" candidate_uuid: "e10fcbe712464d4387591a8a8039c541" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a56be9d05a14dc782220ddfe5cd31c5"
I20260928 14:46:09.488771 18704 raft_consensus.cc:3037] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:09.488859 18571 raft_consensus.cc:3037] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:09.490302 18571 raft_consensus.cc:2454] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e10fcbe712464d4387591a8a8039c541 in term 1.
I20260928 14:46:09.490300 18704 raft_consensus.cc:2454] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e10fcbe712464d4387591a8a8039c541 in term 1.
I20260928 14:46:09.490695 18372 leader_election.cc:302] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e10fcbe712464d4387591a8a8039c541, f235831c56f2496cadbc2415c3d6b44b; no voters:
I20260928 14:46:09.490924 18776 raft_consensus.cc:2781] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:09.491206 18776 raft_consensus.cc:686] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [term 1 LEADER]: Becoming Leader. State: Replica: e10fcbe712464d4387591a8a8039c541, State: Running, Role: LEADER
I20260928 14:46:09.491645 18776 consensus_queue.cc:237] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } }
I20260928 14:46:09.495313 18294 catalog_manager.cc:5495] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 reported cstate change: term changed from 0 to 1, leader changed from <none> to e10fcbe712464d4387591a8a8039c541 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "e10fcbe712464d4387591a8a8039c541" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e10fcbe712464d4387591a8a8039c541" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:09.531729 18763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.527414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59716 (local address 127.13.223.65:37363)
0928 14:46:09.527613 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.527617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.527634 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:09.527958 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.527962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.528040 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.528197 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:09.528483 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.529023 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.530018 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:09.530260 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:09.530384 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:09.530788 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:09.531278 (+ 490us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:09.531426 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.531489 (+ 63us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.531558 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 14:46:09.538815 18704 raft_consensus.cc:1261] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 1 FOLLOWER]: Refusing update from remote peer e10fcbe712464d4387591a8a8039c541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:09.538815 18571 raft_consensus.cc:1261] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 1 FOLLOWER]: Refusing update from remote peer e10fcbe712464d4387591a8a8039c541: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:09.539786 18776 consensus_queue.cc:1035] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f235831c56f2496cadbc2415c3d6b44b" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 42591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:09.540122 18779 consensus_queue.cc:1035] T 8996d2b9c51f4484abf70b8186740683 P e10fcbe712464d4387591a8a8039c541 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a56be9d05a14dc782220ddfe5cd31c5" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:46:09.546342 18618 tablet.cc:2369] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:46:09.551682 18783 mvcc.cc:205] Tried to move back new op lower bound from 7334325328021463040 to 7334325327840796672. Current Snapshot: MvccSnapshot[applied={T|T < 7334325328021463040}]
I20260928 14:46:09.562527 18294 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:09.562786 18294 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:09.565629 18294 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c785faff03b43bdb890cefbb22f0f83: Sending DeleteTablet for 3 replicas of tablet 8996d2b9c51f4484abf70b8186740683
I20260928 14:46:09.566670 18418 tablet_service.cc:1514] Processing DeleteTablet for tablet 8996d2b9c51f4484abf70b8186740683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:09 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:59710
I20260928 14:46:09.566764 18684 tablet_service.cc:1514] Processing DeleteTablet for tablet 8996d2b9c51f4484abf70b8186740683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:09 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:46834
I20260928 14:46:09.566826 18551 tablet_service.cc:1514] Processing DeleteTablet for tablet 8996d2b9c51f4484abf70b8186740683 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:09 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:54822
I20260928 14:46:09.567363 18794 tablet_replica.cc:330] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: stopping tablet replica
I20260928 14:46:09.567538 18793 tablet_replica.cc:330] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: stopping tablet replica
I20260928 14:46:09.567754 18794 raft_consensus.cc:2229] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:09.567901 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18356
I20260928 14:46:09.568141 18794 raft_consensus.cc:2258] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:09.568774 18793 raft_consensus.cc:2229] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:09.569186 18793 raft_consensus.cc:2258] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:09.571171 18793 ts_tablet_manager.cc:1905] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:09.571491 18794 ts_tablet_manager.cc:1905] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:09.574347 18794 ts_tablet_manager.cc:1918] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:09.574347 18793 ts_tablet_manager.cc:1918] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:09.574487 18793 log.cc:1196] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/8996d2b9c51f4484abf70b8186740683
I20260928 14:46:09.574487 18794 log.cc:1196] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/8996d2b9c51f4484abf70b8186740683
I20260928 14:46:09.574908 18794 ts_tablet_manager.cc:1939] T 8996d2b9c51f4484abf70b8186740683 P 6a56be9d05a14dc782220ddfe5cd31c5: Deleting consensus metadata
I20260928 14:46:09.575312 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18489
W20260928 14:46:09.575359 18280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:37363 (error 108)
I20260928 14:46:09.575567 18793 ts_tablet_manager.cc:1939] T 8996d2b9c51f4484abf70b8186740683 P f235831c56f2496cadbc2415c3d6b44b: Deleting consensus metadata
I20260928 14:46:09.577205 18279 catalog_manager.cc:4841] TS f235831c56f2496cadbc2415c3d6b44b (127.13.223.67:42591): tablet 8996d2b9c51f4484abf70b8186740683 (table test-table [id=381572659e21439aa106e8125606202c]) successfully deleted
I20260928 14:46:09.579174 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.578845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:37363 (local address 127.0.0.1:59730)
0928 14:46:09.579006 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.579087 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:37363: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:46:09.579423 18280 catalog_manager.cc:4584] TS e10fcbe712464d4387591a8a8039c541 (127.13.223.65:37363): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8996d2b9c51f4484abf70b8186740683: Network error: Client connection negotiation failed: client connection to 127.13.223.65:37363: connect: Connection refused (error 111)
I20260928 14:46:09.582269 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18622
I20260928 14:46:09.584363 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.584062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35145 (local address 127.0.0.1:54838)
0928 14:46:09.584203 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:09.584274 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:35145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 14:46:09.589402 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18263
2026-09-28T14:46:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2608 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T14:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:09Z Disabled control of system clock
I20260928 14:46:09.623345 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40009
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:40009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:46:09.748476 18799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:09.748834 18799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:09.748879 18799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:09.753947 18799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:09.754045 18799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:09.754062 18799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:09.754082 18799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:09.754101 18799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:09.759531 18799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:40009
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:09.760846 18799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:09.762434 18799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:09.772132 18804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:09.772084 18805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:09.772084 18807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:09.772707 18799 server_base.cc:1034] running on GCE node
I20260928 14:46:09.773453 18799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:09.774844 18799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:09.776038 18799 hybrid_clock.cc:648] HybridClock initialized: now 1790606769776016 us; error 38 us; skew 500 ppm
I20260928 14:46:09.778412 18799 webserver.cc:460] Webserver started at http://127.13.223.126:36285/ using document root <none> and password file <none>
I20260928 14:46:09.779101 18799 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:09.779171 18799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:09.779412 18799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:09.781715 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "ea3e6c6817ad43b49e24156c11a2518d"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5f07f2775a091439399ba2718dd8737f"
server_key_iv: "a79eea2e08e990680f13767547f2b5c7"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.782357 18799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "ea3e6c6817ad43b49e24156c11a2518d"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5f07f2775a091439399ba2718dd8737f"
server_key_iv: "a79eea2e08e990680f13767547f2b5c7"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.786856 18799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:09.789709 18813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:09.791332 18799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:09.791527 18799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "ea3e6c6817ad43b49e24156c11a2518d"
format_stamp: "Formatted at 2026-09-28 14:46:09 on dist-test-slave-gkhx"
server_key: "5f07f2775a091439399ba2718dd8737f"
server_key_iv: "a79eea2e08e990680f13767547f2b5c7"
server_key_version: "encryptionkey@0"
I20260928 14:46:09.791677 18799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:09.796722 18799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:09.797485 18799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:09.797698 18799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:09.806015 18799 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:40009
I20260928 14:46:09.806130 18866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:40009 every 8 connection(s)
I20260928 14:46:09.807144 18799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:09.811168 18867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:09.811264 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18799
I20260928 14:46:09.811463 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:09.811791 14205 external_mini_cluster.cc:1428] Setting key 752dd85d70233e1313b1885ba7f25955
I20260928 14:46:09.818324 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:09.813473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.13.223.126:40009)
0928 14:46:09.814393 (+ 920us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:09.814416 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:09.814459 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:46:09.815204 (+ 745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:09.815223 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:09.815629 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:09.816025 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:09.816966 (+ 941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:09.816971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:09.816985 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:09.816996 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:09.817064 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:09.817081 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:09.817246 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:09.817321 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 14:46:09.817582 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":168,"threads_started":1}
I20260928 14:46:09.819329 18867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: Bootstrap starting.
I20260928 14:46:09.821911 18867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:09.822892 18867 log.cc:824] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:09.825479 18867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: No bootstrap required, opened a new log
I20260928 14:46:09.830722 18867 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } }
I20260928 14:46:09.831049 18867 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:09.831117 18867 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3e6c6817ad43b49e24156c11a2518d, State: Initialized, Role: FOLLOWER
I20260928 14:46:09.831703 18867 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } }
I20260928 14:46:09.831925 18867 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:09.832046 18867 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:09.832176 18867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:09.833359 18867 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } }
I20260928 14:46:09.833776 18867 leader_election.cc:302] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea3e6c6817ad43b49e24156c11a2518d; no voters:
I20260928 14:46:09.834154 18867 leader_election.cc:288] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:09.834306 18872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:09.834627 18872 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [term 1 LEADER]: Becoming Leader. State: Replica: ea3e6c6817ad43b49e24156c11a2518d, State: Running, Role: LEADER
I20260928 14:46:09.835037 18872 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } }
I20260928 14:46:09.835606 18867 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:09.836907 18874 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea3e6c6817ad43b49e24156c11a2518d. Latest consensus state: current_term: 1 leader_uuid: "ea3e6c6817ad43b49e24156c11a2518d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } } }
I20260928 14:46:09.837054 18874 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:09.837373 18873 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea3e6c6817ad43b49e24156c11a2518d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea3e6c6817ad43b49e24156c11a2518d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40009 } } }
I20260928 14:46:09.837489 18873 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:09.837452 18881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:09.841008 18881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:09.848053 18881 catalog_manager.cc:1347] Generated new cluster ID: 6538ab14013340beb253ac37e51803b7
I20260928 14:46:09.848201 18881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:09.874255 18881 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:09.874703 18881 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:09.882264 18881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: Generated new TSK 0
I20260928 14:46:09.883262 18881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:09.887970 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--builtin_ntp_servers=127.13.223.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 14:46:10.015743 18891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:10.016090 18891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:10.016166 18891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:10.020529 18891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:10.020649 18891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:10.020758 18891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:10.026087 18891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:10.027379 18891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:10.028893 18891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:10.039886 18899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.039711 18896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.039902 18897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.040135 18891 server_base.cc:1034] running on GCE node
I20260928 14:46:10.040663 18891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:10.041512 18891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:10.042675 18891 hybrid_clock.cc:648] HybridClock initialized: now 1790606770042656 us; error 39 us; skew 500 ppm
I20260928 14:46:10.044919 18891 webserver.cc:460] Webserver started at http://127.13.223.65:40759/ using document root <none> and password file <none>
I20260928 14:46:10.045608 18891 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:10.045680 18891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:10.045928 18891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:10.048286 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "8d4894d179c041c3a1625a6f2139c88f"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "379d2539abee5122ccc24704d7cf86ea"
server_key_iv: "5cecd12434d22fd01315559edba61877"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.048928 18891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "8d4894d179c041c3a1625a6f2139c88f"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "379d2539abee5122ccc24704d7cf86ea"
server_key_iv: "5cecd12434d22fd01315559edba61877"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.053499 18891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:46:10.057222 18905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.058993 18891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260928 14:46:10.059186 18891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "8d4894d179c041c3a1625a6f2139c88f"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "379d2539abee5122ccc24704d7cf86ea"
server_key_iv: "5cecd12434d22fd01315559edba61877"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.059324 18891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:10.065016 18891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:10.065893 18891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:10.066126 18891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:10.066851 18891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:10.067960 18891 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:10.068041 18891 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.068123 18891 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:10.068177 18891 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.078521 18891 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:37973
I20260928 14:46:10.078567 19019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:37973 every 8 connection(s)
I20260928 14:46:10.079613 18891 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:10.086001 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 18891
I20260928 14:46:10.086179 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:10.086524 14205 external_mini_cluster.cc:1428] Setting key 1db70f1381c47b08e6e86d2efde5acc0
I20260928 14:46:10.086987 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.081810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:57123 (local address 127.13.223.126:40009)
0928 14:46:10.082013 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.082018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.083347 (+ 1329us) server_negotiation.cc:407] Connection header received
0928 14:46:10.084791 (+ 1444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.084794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.084840 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.084925 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.086411 (+ 1486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.086415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.086419 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.086435 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.086469 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.086479 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.086619 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.086778 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.086831 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260928 14:46:10.088408 19022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.082299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40009 (local address 127.13.223.65:57123)
0928 14:46:10.083149 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.083200 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.084511 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.085122 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.085130 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.085235 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.085982 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.085984 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.086037 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.086149 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.086603 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.086614 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.086620 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.086834 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.087337 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":203,"threads_started":1}
I20260928 14:46:10.089036 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--builtin_ntp_servers=127.13.223.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:46:10.089962 19020 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40009
I20260928 14:46:10.090247 19020 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:10.090853 19020 heartbeater.cc:511] Master 127.13.223.126:40009 requested a full tablet report, sending...
I20260928 14:46:10.092839 18831 ts_manager.cc:194] Registered new tserver with Master: 8d4894d179c041c3a1625a6f2139c88f (127.13.223.65:37973)
W20260928 14:46:10.209043 19023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:10.209349 19023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:10.209398 19023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:10.213650 19023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:10.213761 19023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:10.213826 19023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:10.218773 19023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:10.219960 19023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:10.221392 19023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:10.232292 19028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.232292 19029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.232321 19031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.232350 19023 server_base.cc:1034] running on GCE node
I20260928 14:46:10.233062 19023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:10.233830 19023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:10.235198 19023 hybrid_clock.cc:648] HybridClock initialized: now 1790606770235158 us; error 58 us; skew 500 ppm
I20260928 14:46:10.237636 19023 webserver.cc:460] Webserver started at http://127.13.223.66:45567/ using document root <none> and password file <none>
I20260928 14:46:10.238261 19023 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:10.238330 19023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:10.238514 19023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:10.240743 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "e98429aa00d74fc68bb9759fbf967428"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "3bc62fbd0cfc66c74a7edf3e5ba71d61"
server_key_iv: "98ee40ddd9ff41c6970895f68b7c6c5b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.241308 19023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "e98429aa00d74fc68bb9759fbf967428"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "3bc62fbd0cfc66c74a7edf3e5ba71d61"
server_key_iv: "98ee40ddd9ff41c6970895f68b7c6c5b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.245589 19023 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:46:10.248709 19037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.250352 19023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:10.250504 19023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "e98429aa00d74fc68bb9759fbf967428"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "3bc62fbd0cfc66c74a7edf3e5ba71d61"
server_key_iv: "98ee40ddd9ff41c6970895f68b7c6c5b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.250702 19023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:10.255905 19023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:10.256582 19023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:10.256740 19023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:10.257295 19023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:10.258265 19023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:10.258315 19023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.258358 19023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:10.258374 19023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.269176 19023 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:36037
I20260928 14:46:10.269224 19151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:36037 every 8 connection(s)
I20260928 14:46:10.270644 19023 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:10.276849 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19023
I20260928 14:46:10.276970 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:10.277247 14205 external_mini_cluster.cc:1428] Setting key 11ec059726d64ced6054f514718d374b
I20260928 14:46:10.280697 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.273033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:53389 (local address 127.13.223.126:40009)
0928 14:46:10.273190 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.273195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.274929 (+ 1734us) server_negotiation.cc:407] Connection header received
0928 14:46:10.278186 (+ 3257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.278190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.278241 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.278292 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.280154 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.280158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.280161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.280175 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.280199 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.280209 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.280271 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.280502 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.280549 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:46:10.282444 19154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.273645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40009 (local address 127.13.223.66:53389)
0928 14:46:10.274634 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.274745 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.277888 (+ 3143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.278748 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.278758 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.278890 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.279812 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.279814 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.279875 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.279974 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.280348 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.280356 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.280361 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.281081 (+ 720us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.281381 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":165,"threads_started":1}
I20260928 14:46:10.283944 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--builtin_ntp_servers=127.13.223.84:41983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 14:46:10.286995 19152 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40009
I20260928 14:46:10.287315 19152 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:10.287835 19152 heartbeater.cc:511] Master 127.13.223.126:40009 requested a full tablet report, sending...
I20260928 14:46:10.289198 18831 ts_manager.cc:194] Registered new tserver with Master: e98429aa00d74fc68bb9759fbf967428 (127.13.223.66:36037)
W20260928 14:46:10.402743 19155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:10.403110 19155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:10.403208 19155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:10.407562 19155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:10.407660 19155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:10.407752 19155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:10.412887 19155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40009
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:10.414108 19155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:10.415621 19155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:10.426729 19160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.426771 19163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.426832 19155 server_base.cc:1034] running on GCE node
W20260928 14:46:10.426743 19161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.427572 19155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:10.428367 19155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:10.429577 19155 hybrid_clock.cc:648] HybridClock initialized: now 1790606770429558 us; error 35 us; skew 500 ppm
I20260928 14:46:10.431855 19155 webserver.cc:460] Webserver started at http://127.13.223.67:43887/ using document root <none> and password file <none>
I20260928 14:46:10.432490 19155 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:10.432588 19155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:10.432833 19155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:10.435107 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "a254a74557324ab087b820274b9c98d0"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "0437d213ab416a602237867346b3bca1"
server_key_iv: "bcf451097459e952c2a16646a542877d"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.435776 19155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "a254a74557324ab087b820274b9c98d0"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "0437d213ab416a602237867346b3bca1"
server_key_iv: "bcf451097459e952c2a16646a542877d"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.440325 19155 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:10.443537 19169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.445156 19155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:46:10.445324 19155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "a254a74557324ab087b820274b9c98d0"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "0437d213ab416a602237867346b3bca1"
server_key_iv: "bcf451097459e952c2a16646a542877d"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.445466 19155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:10.450430 19155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:10.451140 19155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:10.451336 19155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:10.451994 19155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:10.452976 19155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:10.453055 19155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.453130 19155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:10.453182 19155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.463004 19155 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40133
I20260928 14:46:10.463270 19283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40133 every 8 connection(s)
I20260928 14:46:10.464154 19155 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:10.472590 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.466598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:42945 (local address 127.13.223.126:40009)
0928 14:46:10.466895 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.466899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.468427 (+ 1528us) server_negotiation.cc:407] Connection header received
0928 14:46:10.469847 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.469851 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.469894 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.469994 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.471792 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.471796 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.471799 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.471816 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.471843 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.471853 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.471963 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.472360 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.472425 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194}
I20260928 14:46:10.472609 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19155
I20260928 14:46:10.472721 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:10.473042 14205 external_mini_cluster.cc:1428] Setting key 2e1df839816b404a081dac596c99968b
I20260928 14:46:10.473860 19286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.467135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40009 (local address 127.13.223.67:42945)
0928 14:46:10.468189 (+ 1054us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.468237 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.469554 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.470209 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.470217 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.470323 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.471435 (+ 1112us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.471438 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.471491 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.471583 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.471994 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.472002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.472027 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.472245 (+ 218us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.472504 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":170,"threads_started":1}
I20260928 14:46:10.475486 19284 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40009
I20260928 14:46:10.475827 19284 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:10.476380 19284 heartbeater.cc:511] Master 127.13.223.126:40009 requested a full tablet report, sending...
I20260928 14:46:10.477645 18831 ts_manager.cc:194] Registered new tserver with Master: a254a74557324ab087b820274b9c98d0 (127.13.223.67:40133)
I20260928 14:46:10.489045 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:46:10.493574 18870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.491574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43022 (local address 127.13.223.126:40009)
0928 14:46:10.491758 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.491763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.491907 (+ 144us) server_negotiation.cc:407] Connection header received
0928 14:46:10.492114 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.492118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.492156 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.492303 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.493114 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.493118 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.493120 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.493140 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.493161 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.493168 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.493267 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.493361 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.493424 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 14:46:10.497385 18831 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:10.501037 18831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:10.514655 19295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.511406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40133 (local address 127.0.0.1:34992)
0928 14:46:10.512104 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.512163 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.512268 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.513062 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.513068 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.513147 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.513748 (+ 601us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.513752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.513771 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.513853 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.514268 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.514277 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.514283 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.514398 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.514472 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":72,"threads_started":1}
I20260928 14:46:10.515064 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.512215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:37973 (local address 127.0.0.1:50328)
0928 14:46:10.512872 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.512887 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.512966 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.513789 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.513791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.513805 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.514374 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.514379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.514389 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.514414 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.514828 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.514831 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.514834 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.514904 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.514940 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":82,"threads_started":1}
I20260928 14:46:10.515080 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.511531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34992 (local address 127.13.223.67:40133)
0928 14:46:10.512569 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.512582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.512608 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:46:10.512664 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.512670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.512878 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.513069 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.514021 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.514030 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.514048 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.514059 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.514125 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.514174 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.514300 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.514537 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.514651 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":901,"thread_start_us":174,"threads_started":1}
I20260928 14:46:10.515498 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.512392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50328 (local address 127.13.223.65:37973)
0928 14:46:10.512808 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.512821 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.513038 (+ 217us) server_negotiation.cc:407] Connection header received
0928 14:46:10.513183 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.513188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.513318 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.513442 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.514581 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.514587 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.514596 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.514608 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.514647 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.514681 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.514781 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.515273 (+ 492us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.515347 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":153,"threads_started":1}
I20260928 14:46:10.515610 19299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.511584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:36037 (local address 127.0.0.1:37772)
0928 14:46:10.512469 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.512482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.512552 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.513800 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.513806 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.513828 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.514485 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.514487 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.514498 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.514526 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.515405 (+ 879us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.515408 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.515412 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.515468 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.515501 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":137,"threads_started":1}
I20260928 14:46:10.516163 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.511729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37772 (local address 127.13.223.66:36037)
0928 14:46:10.513026 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.513038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.513061 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:46:10.513117 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.513126 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.513310 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.514343 (+ 1033us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.515204 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.515209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.515218 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.515227 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.515264 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.515287 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.515878 (+ 591us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.515931 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.516004 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1186,"thread_start_us":88,"threads_started":1}
I20260928 14:46:10.518527 19086 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 (DEFAULT_TABLE table=test-table [id=d3ceb9a9a3a3492cbffa2858cdb64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:10.518635 19218 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 (DEFAULT_TABLE table=test-table [id=d3ceb9a9a3a3492cbffa2858cdb64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:10.518661 18954 tablet_service.cc:1467] Processing CreateTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 (DEFAULT_TABLE table=test-table [id=d3ceb9a9a3a3492cbffa2858cdb64ca5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:10.519824 19086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd465f4b80b4865bd7b452ca9a74207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:10.519824 18954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd465f4b80b4865bd7b452ca9a74207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:10.519824 19218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dd465f4b80b4865bd7b452ca9a74207. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:10.526654 19301 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Bootstrap starting.
I20260928 14:46:10.528299 19303 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Bootstrap starting.
I20260928 14:46:10.529489 19301 tablet_bootstrap.cc:654] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:10.530282 19303 tablet_bootstrap.cc:654] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:10.530354 19301 log.cc:824] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:10.530970 19302 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Bootstrap starting.
I20260928 14:46:10.531107 19303 log.cc:824] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:10.532711 19301 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: No bootstrap required, opened a new log
I20260928 14:46:10.532922 19301 ts_tablet_manager.cc:1397] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:46:10.533011 19302 tablet_bootstrap.cc:654] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:10.533182 19303 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: No bootstrap required, opened a new log
I20260928 14:46:10.533351 19303 ts_tablet_manager.cc:1397] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 14:46:10.533710 19302 log.cc:824] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:10.537269 19302 tablet_bootstrap.cc:492] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: No bootstrap required, opened a new log
I20260928 14:46:10.537528 19302 ts_tablet_manager.cc:1397] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20260928 14:46:10.538000 19301 raft_consensus.cc:348] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.538000 19303 raft_consensus.cc:348] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.538303 19303 raft_consensus.cc:374] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:10.538303 19301 raft_consensus.cc:374] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:10.538363 19301 raft_consensus.cc:729] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e98429aa00d74fc68bb9759fbf967428, State: Initialized, Role: FOLLOWER
I20260928 14:46:10.538372 19303 raft_consensus.cc:729] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d4894d179c041c3a1625a6f2139c88f, State: Initialized, Role: FOLLOWER
I20260928 14:46:10.538885 19303 consensus_queue.cc:260] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.538885 19301 consensus_queue.cc:260] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.539662 19020 heartbeater.cc:503] Master 127.13.223.126:40009 was elected leader, sending a full tablet report...
I20260928 14:46:10.540004 19303 ts_tablet_manager.cc:1428] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 14:46:10.540004 19301 ts_tablet_manager.cc:1428] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:10.540323 19152 heartbeater.cc:503] Master 127.13.223.126:40009 was elected leader, sending a full tablet report...
I20260928 14:46:10.542266 19302 raft_consensus.cc:348] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.542610 19302 raft_consensus.cc:374] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:10.542673 19302 raft_consensus.cc:729] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a254a74557324ab087b820274b9c98d0, State: Initialized, Role: FOLLOWER
I20260928 14:46:10.543215 19302 consensus_queue.cc:260] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.543999 19284 heartbeater.cc:503] Master 127.13.223.126:40009 was elected leader, sending a full tablet report...
I20260928 14:46:10.544324 19302 ts_tablet_manager.cc:1428] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Time spent starting tablet: real 0.007s user 0.002s sys 0.006s
W20260928 14:46:10.581003 19021 tablet.cc:2369] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:46:10.627491 19307 raft_consensus.cc:482] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:10.627720 19307 raft_consensus.cc:504] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.628993 19307 leader_election.cc:288] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a254a74557324ab087b820274b9c98d0 (127.13.223.67:40133), e98429aa00d74fc68bb9759fbf967428 (127.13.223.66:36037)
I20260928 14:46:10.631825 19298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.629439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:54743 (local address 127.13.223.66:36037)
0928 14:46:10.629577 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.629584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.629674 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:46:10.629963 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.629966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.630006 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.630088 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.631284 (+ 1196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.631288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.631291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.631304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.631328 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.631339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.631446 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.631651 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.631691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:10.631825 19296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.629695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:58417 (local address 127.13.223.67:40133)
0928 14:46:10.629855 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.629859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.629872 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:46:10.629913 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.629916 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.629957 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.630053 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.631284 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.631288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.631291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.631304 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.631328 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.631339 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.631446 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.631635 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.631684 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:46:10.631917 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.629330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:36037 (local address 127.13.223.65:54743)
0928 14:46:10.629567 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.629583 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.629707 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.630169 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.630173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.630195 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.631036 (+ 841us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.631039 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.631054 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.631091 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.631496 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.631501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.631506 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.631630 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.631719 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":26,"thread_start_us":72,"threads_started":1}
I20260928 14:46:10.631917 19311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.629366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40133 (local address 127.13.223.65:58417)
0928 14:46:10.629678 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.629689 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:10.629746 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:10.630169 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:10.630173 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:10.630195 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:10.631058 (+ 863us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:10.631060 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:10.631066 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:10.631094 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:10.631496 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:10.631501 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:10.631506 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:46:10.631630 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:46:10.631702 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":65,"threads_started":1}
I20260928 14:46:10.632812 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd465f4b80b4865bd7b452ca9a74207" candidate_uuid: "8d4894d179c041c3a1625a6f2139c88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e98429aa00d74fc68bb9759fbf967428" is_pre_election: true
I20260928 14:46:10.632810 19238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd465f4b80b4865bd7b452ca9a74207" candidate_uuid: "8d4894d179c041c3a1625a6f2139c88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a254a74557324ab087b820274b9c98d0" is_pre_election: true
I20260928 14:46:10.633149 19238 raft_consensus.cc:2454] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d4894d179c041c3a1625a6f2139c88f in term 0.
I20260928 14:46:10.633149 19106 raft_consensus.cc:2454] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d4894d179c041c3a1625a6f2139c88f in term 0.
I20260928 14:46:10.633736 18909 leader_election.cc:302] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d4894d179c041c3a1625a6f2139c88f, a254a74557324ab087b820274b9c98d0, e98429aa00d74fc68bb9759fbf967428; no voters:
I20260928 14:46:10.634099 19307 raft_consensus.cc:2781] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:10.634198 19307 raft_consensus.cc:482] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:10.634260 19307 raft_consensus.cc:3037] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:10.635937 19307 raft_consensus.cc:504] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.636421 19307 leader_election.cc:288] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [CANDIDATE]: Term 1 election: Requested vote from peers a254a74557324ab087b820274b9c98d0 (127.13.223.67:40133), e98429aa00d74fc68bb9759fbf967428 (127.13.223.66:36037)
I20260928 14:46:10.636854 19238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd465f4b80b4865bd7b452ca9a74207" candidate_uuid: "8d4894d179c041c3a1625a6f2139c88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a254a74557324ab087b820274b9c98d0"
I20260928 14:46:10.636895 19106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dd465f4b80b4865bd7b452ca9a74207" candidate_uuid: "8d4894d179c041c3a1625a6f2139c88f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e98429aa00d74fc68bb9759fbf967428"
I20260928 14:46:10.637017 19106 raft_consensus.cc:3037] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:10.637022 19238 raft_consensus.cc:3037] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:10.638213 19106 raft_consensus.cc:2454] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d4894d179c041c3a1625a6f2139c88f in term 1.
I20260928 14:46:10.638363 19238 raft_consensus.cc:2454] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d4894d179c041c3a1625a6f2139c88f in term 1.
I20260928 14:46:10.638618 18907 leader_election.cc:302] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d4894d179c041c3a1625a6f2139c88f, e98429aa00d74fc68bb9759fbf967428; no voters:
I20260928 14:46:10.638849 19307 raft_consensus.cc:2781] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:10.639093 19307 raft_consensus.cc:686] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 1 LEADER]: Becoming Leader. State: Replica: 8d4894d179c041c3a1625a6f2139c88f, State: Running, Role: LEADER
I20260928 14:46:10.639479 19307 consensus_queue.cc:237] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } }
I20260928 14:46:10.643163 18831 catalog_manager.cc:5495] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d4894d179c041c3a1625a6f2139c88f (127.13.223.65). New cstate: current_term: 1 leader_uuid: "8d4894d179c041c3a1625a6f2139c88f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d4894d179c041c3a1625a6f2139c88f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 37973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:10.673517 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.671412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50336 (local address 127.13.223.65:37973)
0928 14:46:10.671571 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.671575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.671592 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:10.671764 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.671767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.671838 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.672014 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.672912 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.672915 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.672917 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.672929 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.672952 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.672960 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.673141 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.673323 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.673380 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:46:10.680330 19106 raft_consensus.cc:1261] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 1 FOLLOWER]: Refusing update from remote peer 8d4894d179c041c3a1625a6f2139c88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:10.680330 19238 raft_consensus.cc:1261] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 1 FOLLOWER]: Refusing update from remote peer 8d4894d179c041c3a1625a6f2139c88f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:10.681126 19307 consensus_queue.cc:1035] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a254a74557324ab087b820274b9c98d0" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:10.681412 19312 consensus_queue.cc:1035] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e98429aa00d74fc68bb9759fbf967428" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 36037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:10.692152 19317 mvcc.cc:205] Tried to move back new op lower bound from 7334325332698382336 to 7334325332542328832. Current Snapshot: MvccSnapshot[applied={T|T < 7334325332698382336 or (T in {7334325332698382336})}]
I20260928 14:46:10.702284 18829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:10.702548 18829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:10.705866 18829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea3e6c6817ad43b49e24156c11a2518d: Sending DeleteTablet for 3 replicas of tablet 9dd465f4b80b4865bd7b452ca9a74207
I20260928 14:46:10.706933 19218 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:10 UTC) from {username='slave'} at 127.0.0.1:34992
I20260928 14:46:10.706956 19086 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:10 UTC) from {username='slave'} at 127.0.0.1:37772
I20260928 14:46:10.707065 18954 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dd465f4b80b4865bd7b452ca9a74207 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:10 UTC) from {username='slave'} at 127.0.0.1:50328
I20260928 14:46:10.707589 19328 tablet_replica.cc:330] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f: stopping tablet replica
I20260928 14:46:10.707962 19327 tablet_replica.cc:330] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: stopping tablet replica
I20260928 14:46:10.708106 19328 raft_consensus.cc:2229] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:10.708496 19327 raft_consensus.cc:2229] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:10.708606 19328 raft_consensus.cc:2258] T 9dd465f4b80b4865bd7b452ca9a74207 P 8d4894d179c041c3a1625a6f2139c88f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:10.708745 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18891
I20260928 14:46:10.709050 19326 tablet_replica.cc:330] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: stopping tablet replica
I20260928 14:46:10.709528 19326 raft_consensus.cc:2229] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:10.709882 19326 raft_consensus.cc:2258] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:10.711578 19327 raft_consensus.cc:2258] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:10.712157 19326 ts_tablet_manager.cc:1905] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:10.713546 19327 ts_tablet_manager.cc:1905] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:10.715940 19326 ts_tablet_manager.cc:1918] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:10.716073 19326 log.cc:1196] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/9dd465f4b80b4865bd7b452ca9a74207
I20260928 14:46:10.716298 19327 ts_tablet_manager.cc:1918] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:10.716439 19327 log.cc:1196] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/9dd465f4b80b4865bd7b452ca9a74207
I20260928 14:46:10.716436 19326 ts_tablet_manager.cc:1939] T 9dd465f4b80b4865bd7b452ca9a74207 P a254a74557324ab087b820274b9c98d0: Deleting consensus metadata
I20260928 14:46:10.716747 19327 ts_tablet_manager.cc:1939] T 9dd465f4b80b4865bd7b452ca9a74207 P e98429aa00d74fc68bb9759fbf967428: Deleting consensus metadata
W20260928 14:46:10.717269 18816 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:37973 (error 108)
I20260928 14:46:10.718523 19297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.718150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:37973 (local address 127.0.0.1:35932)
0928 14:46:10.718307 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:10.718403 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:37973: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:46:10.718530 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19023
I20260928 14:46:10.718595 18814 catalog_manager.cc:4841] TS e98429aa00d74fc68bb9759fbf967428 (127.13.223.66:36037): tablet 9dd465f4b80b4865bd7b452ca9a74207 (table test-table [id=d3ceb9a9a3a3492cbffa2858cdb64ca5]) successfully deleted
I20260928 14:46:10.718595 18816 catalog_manager.cc:4841] TS a254a74557324ab087b820274b9c98d0 (127.13.223.67:40133): tablet 9dd465f4b80b4865bd7b452ca9a74207 (table test-table [id=d3ceb9a9a3a3492cbffa2858cdb64ca5]) successfully deleted
W20260928 14:46:10.718937 18816 catalog_manager.cc:4584] TS 8d4894d179c041c3a1625a6f2139c88f (127.13.223.65:37973): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dd465f4b80b4865bd7b452ca9a74207: Network error: Client connection negotiation failed: client connection to 127.13.223.65:37973: connect: Connection refused (error 111)
I20260928 14:46:10.725162 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19155
I20260928 14:46:10.731168 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 18799
2026-09-28T14:46:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1140 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 14:46:10.754865 14205 mini_oidc.cc:152] Starting JWKS server
I20260928 14:46:10.756528 14205 webserver.cc:460] Webserver started at https://127.0.0.1:41015/ using document root <none> and password file <none>
I20260928 14:46:10.756685 14205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:46:10.756922 14205 webserver.cc:460] Webserver started at http://127.0.0.1:41647/ using document root <none> and password file <none>
2026-09-28T14:46:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:10Z Disabled control of system clock
I20260928 14:46:10.768649 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43937
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:42849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:43937
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41015/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/testchainca.cert with env {}
W20260928 14:46:10.902365 19335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:10.902732 19335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:10.902776 19335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:10.907405 19335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:10.907485 19335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:10.907506 19335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:10.907526 19335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:10.907542 19335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:46:10.907559 19335 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:46:10.907572 19335 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41015/jwks.json
I20260928 14:46:10.913475 19335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:43937
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41015/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:10.914644 19335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:10.915663 19335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:10.925479 19340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.925616 19335 server_base.cc:1034] running on GCE node
W20260928 14:46:10.925580 19343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:10.925943 19341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:10.926887 19335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:10.928254 19335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:10.929461 19335 hybrid_clock.cc:648] HybridClock initialized: now 1790606770929446 us; error 38 us; skew 500 ppm
I20260928 14:46:10.931999 19335 webserver.cc:460] Webserver started at http://127.13.223.126:36607/ using document root <none> and password file <none>
I20260928 14:46:10.932940 19335 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:10.933035 19335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:10.933308 19335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:10.935770 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "a6bf93bdcb824670bb3731b93e3b0112"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "81147430c2c9d48a51f71995dca03595"
server_key_iv: "b6907eb9eb3d7fa986803623511b431b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.936463 19335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "a6bf93bdcb824670bb3731b93e3b0112"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "81147430c2c9d48a51f71995dca03595"
server_key_iv: "b6907eb9eb3d7fa986803623511b431b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.941056 19335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:10.944625 19349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:10.946442 19335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:10.946741 19335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "a6bf93bdcb824670bb3731b93e3b0112"
format_stamp: "Formatted at 2026-09-28 14:46:10 on dist-test-slave-gkhx"
server_key: "81147430c2c9d48a51f71995dca03595"
server_key_iv: "b6907eb9eb3d7fa986803623511b431b"
server_key_version: "encryptionkey@0"
I20260928 14:46:10.946907 19335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:10.969705 19335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:10.970760 19335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:10.971012 19335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:10.980273 19335 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:43937
I20260928 14:46:10.980270 19401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:43937 every 8 connection(s)
I20260928 14:46:10.981483 19335 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:10.985263 19402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:10.988037 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19335
I20260928 14:46:10.988180 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:10.988490 14205 external_mini_cluster.cc:1428] Setting key ab3e5e1ae8e3fea07bdd33bff68a1fbf
I20260928 14:46:10.992412 19402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112: Bootstrap starting.
I20260928 14:46:10.995085 19402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:10.995937 19402 log.cc:824] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:10.998699 19402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112: No bootstrap required, opened a new log
I20260928 14:46:10.999924 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:10.990236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.13.223.126:43937)
0928 14:46:10.991016 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:10.991034 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:10.991078 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:46:10.991803 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:10.991823 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:10.992321 (+ 498us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:10.992741 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:10.993322 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:10.994705 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:10.995967 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:10.996599 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:10.996803 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:10.997877 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:10.997883 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:10.997899 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:10.998344 (+ 445us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:10.998409 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:10.998465 (+ 56us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:10.998617 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:10.998801 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 14:46:10.999144 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":167,"threads_started":1}
I20260928 14:46:11.003340 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:46:11.003111 19402 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } }
I20260928 14:46:11.003504 19402 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:11.003589 19402 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6bf93bdcb824670bb3731b93e3b0112, State: Initialized, Role: FOLLOWER
I20260928 14:46:11.004130 19402 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } }
I20260928 14:46:11.004343 19402 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:11.004421 19402 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:11.004534 19402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:11.005766 19402 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } }
I20260928 14:46:11.006167 19402 leader_election.cc:302] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6bf93bdcb824670bb3731b93e3b0112; no voters:
I20260928 14:46:11.006536 19402 leader_election.cc:288] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:11.006816 19407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:11.007145 19407 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [term 1 LEADER]: Becoming Leader. State: Replica: a6bf93bdcb824670bb3731b93e3b0112, State: Running, Role: LEADER
I20260928 14:46:11.007637 19407 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } }
I20260928 14:46:11.008292 19402 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:11.009716 19408 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6bf93bdcb824670bb3731b93e3b0112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } } }
I20260928 14:46:11.009851 19408 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.010231 19409 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6bf93bdcb824670bb3731b93e3b0112. Latest consensus state: current_term: 1 leader_uuid: "a6bf93bdcb824670bb3731b93e3b0112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6bf93bdcb824670bb3731b93e3b0112" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43937 } } }
I20260928 14:46:11.010344 19409 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.010331 19416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:11.013770 19416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:11.020910 19416 catalog_manager.cc:1347] Generated new cluster ID: 5d1f8c6bcf004f46bcc9db00ca2e37b2
I20260928 14:46:11.021047 19416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:11.036716 19416 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:11.037983 19416 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:11.052175 19416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6bf93bdcb824670bb3731b93e3b0112: Generated new TSK 0
I20260928 14:46:11.053263 19416 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:46:11.097472 19432 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 14:46:11.097617 19432 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:46:11.098088 19405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:11.072364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43300 (local address 127.13.223.126:43937)
0928 14:46:11.072513 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:11.072518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:11.072676 (+ 158us) server_negotiation.cc:407] Connection header received
0928 14:46:11.073037 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:11.073041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:11.073103 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:11.073214 (+ 111us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:46:11.073499 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.074083 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.075095 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.075357 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.075469 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.075887 (+ 418us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:46:11.097501 (+ 21614us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:46:11.097903 (+ 402us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 14:46:11.098196 19405 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43300: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 14:46:11.099521 19431 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:11.072167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43937 (local address 127.0.0.1:43300)
0928 14:46:11.072569 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:11.072590 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:11.072766 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:11.073234 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:11.073237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:11.073246 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:46:11.073379 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.073385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.074210 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.074214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.074959 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.074969 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.075496 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.075500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.075590 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.075647 (+ 57us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:46:11.097943 (+ 22296us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 14:46:11.098247 (+ 304us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.126:43937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.13.223.126:43937: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41015/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 14:46:11.100642 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19335
2026-09-28T14:46:11Z chronyd exiting
I20260928 14:46:11.139425 14205 test_util.cc:175] -----------------------------------------------
I20260928 14:46:11.139544 14205 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790606739990168-14205-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (386 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 14:46:11.140769 14205 mini_oidc.cc:152] Starting JWKS server
I20260928 14:46:11.141765 14205 webserver.cc:460] Webserver started at https://127.0.0.1:37959/ using document root <none> and password file <none>
I20260928 14:46:11.141860 14205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:46:11.142169 14205 webserver.cc:460] Webserver started at http://127.0.0.1:36411/ using document root <none> and password file <none>
2026-09-28T14:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:11Z Disabled control of system clock
I20260928 14:46:11.153220 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46281
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:38237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46281
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37959/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/testchainca.cert with env {}
W20260928 14:46:11.273348 19438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:11.273732 19438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:11.273809 19438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:11.278421 19438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:11.278554 19438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:11.278604 19438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:11.278636 19438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:11.278672 19438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:46:11.278712 19438 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:46:11.278738 19438 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37959/jwks.json
I20260928 14:46:11.284811 19438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46281
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37959/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:11.286114 19438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:11.287340 19438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:11.296952 19443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:11.297080 19438 server_base.cc:1034] running on GCE node
W20260928 14:46:11.296967 19446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:11.296986 19444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:11.298004 19438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:11.299508 19438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:11.300767 19438 hybrid_clock.cc:648] HybridClock initialized: now 1790606771300715 us; error 69 us; skew 500 ppm
I20260928 14:46:11.303092 19438 webserver.cc:460] Webserver started at http://127.13.223.126:42253/ using document root <none> and password file <none>
I20260928 14:46:11.303678 19438 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:11.303743 19438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:11.303925 19438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:11.306259 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "55076ee0c8ae4e3c89ba4658a9bfc592"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "cfed24751769a42860ae47b6dbe0f939"
server_key_iv: "f622eb2384b547cf9dcb19341ff12c83"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.306932 19438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "55076ee0c8ae4e3c89ba4658a9bfc592"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "cfed24751769a42860ae47b6dbe0f939"
server_key_iv: "f622eb2384b547cf9dcb19341ff12c83"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.311450 19438 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:46:11.314466 19452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:11.316111 19438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:11.316366 19438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "55076ee0c8ae4e3c89ba4658a9bfc592"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "cfed24751769a42860ae47b6dbe0f939"
server_key_iv: "f622eb2384b547cf9dcb19341ff12c83"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.316532 19438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:11.332055 19438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:11.332893 19438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:11.333124 19438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:11.340819 19438 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:46281
I20260928 14:46:11.340845 19504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:46281 every 8 connection(s)
I20260928 14:46:11.341908 19438 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:11.345566 19505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:11.350835 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19438
I20260928 14:46:11.351014 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:11.351310 14205 external_mini_cluster.cc:1428] Setting key e5c70e5f3d438e024a846d9cf1cad313
I20260928 14:46:11.352005 19505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592: Bootstrap starting.
I20260928 14:46:11.354527 19505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:11.355489 19505 log.cc:824] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:11.357837 19505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592: No bootstrap required, opened a new log
I20260928 14:46:11.362066 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:11.352947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59756 (local address 127.13.223.126:46281)
0928 14:46:11.353662 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:11.353678 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:11.353715 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:46:11.354424 (+ 709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:11.354444 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:11.355022 (+ 578us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:11.355330 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:11.355769 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.356765 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.358053 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.358770 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.358949 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.360033 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:11.360039 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:11.360053 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:11.360505 (+ 452us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:11.360571 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:11.360593 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:11.360680 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:11.360849 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:46:11.361255 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":170,"threads_started":1}
I20260928 14:46:11.362178 19505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } }
I20260928 14:46:11.362509 19505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:11.362623 19505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55076ee0c8ae4e3c89ba4658a9bfc592, State: Initialized, Role: FOLLOWER
I20260928 14:46:11.363142 19505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } }
I20260928 14:46:11.363317 19505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:11.363385 19505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:11.363479 19505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:11.364748 19505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } }
I20260928 14:46:11.365142 19505 leader_election.cc:302] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55076ee0c8ae4e3c89ba4658a9bfc592; no voters:
I20260928 14:46:11.365451 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:46:11.365526 19505 leader_election.cc:288] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:11.365639 19510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:11.365844 19510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [term 1 LEADER]: Becoming Leader. State: Replica: 55076ee0c8ae4e3c89ba4658a9bfc592, State: Running, Role: LEADER
I20260928 14:46:11.366258 19510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } }
I20260928 14:46:11.367301 19505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:11.368273 19511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } } }
I20260928 14:46:11.368273 19512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55076ee0c8ae4e3c89ba4658a9bfc592. Latest consensus state: current_term: 1 leader_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55076ee0c8ae4e3c89ba4658a9bfc592" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46281 } } }
I20260928 14:46:11.368585 19512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.368585 19511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.369688 19519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:11.372776 19519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:11.379053 19519 catalog_manager.cc:1347] Generated new cluster ID: c84688e266594d5083b7aebfe1d50132
I20260928 14:46:11.379151 19519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:11.412639 19519 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:11.413687 19519 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:11.427677 19519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55076ee0c8ae4e3c89ba4658a9bfc592: Generated new TSK 0
I20260928 14:46:11.428651 19519 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 14:46:11.467283 19535 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 14:46:11.467456 19535 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 14:46:11.468379 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:11.444340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.13.223.126:46281)
0928 14:46:11.444528 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:11.444535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:11.444765 (+ 230us) server_negotiation.cc:407] Connection header received
0928 14:46:11.444992 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:11.444995 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:11.445047 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:11.445178 (+ 131us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:46:11.445431 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.446010 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.447049 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.447283 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.447397 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.447728 (+ 331us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:46:11.467850 (+ 20122us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:46:11.468191 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 14:46:11.468573 19534 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:11.444253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46281 (local address 127.0.0.1:59762)
0928 14:46:11.444670 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:11.444683 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:11.444829 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:11.445176 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:11.445180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:11.445189 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:46:11.445310 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.445324 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.446156 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.446160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.446910 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.446919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.447409 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.447412 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.447509 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.447533 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:46:11.468093 (+ 20560us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 14:46:11.468301 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.126:46281: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":137,"threads_started":1}
W20260928 14:46:11.468559 19508 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59762: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37959/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260928 14:46:11.469394 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19438
2026-09-28T14:46:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (369 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 14:46:11.509778 14205 mini_oidc.cc:152] Starting JWKS server
I20260928 14:46:11.510931 14205 webserver.cc:460] Webserver started at https://127.0.0.1:43865/ using document root <none> and password file <none>
I20260928 14:46:11.511057 14205 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 14:46:11.511297 14205 webserver.cc:460] Webserver started at http://127.0.0.1:37057/ using document root <none> and password file <none>
2026-09-28T14:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:11Z Disabled control of system clock
I20260928 14:46:11.521550 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34897
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43865/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:46:11.650455 19541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:11.650928 19541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:11.651034 19541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:11.655783 19541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:11.655920 19541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:11.655952 19541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:11.655997 19541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:11.656034 19541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:46:11.656075 19541 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:46:11.656111 19541 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43865/jwks.json
I20260928 14:46:11.661821 19541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43865/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:11.663170 19541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:11.664736 19541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:11.674746 19549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:11.674734 19546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:11.674734 19547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:11.675097 19541 server_base.cc:1034] running on GCE node
I20260928 14:46:11.675953 19541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:11.677363 19541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:11.678624 19541 hybrid_clock.cc:648] HybridClock initialized: now 1790606771678594 us; error 37 us; skew 500 ppm
I20260928 14:46:11.680861 19541 webserver.cc:460] Webserver started at http://127.13.223.126:36049/ using document root <none> and password file <none>
I20260928 14:46:11.681550 19541 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:11.681625 19541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:11.681864 19541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:11.684204 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "b0a29f44e4a8421ea78ed2fa249c71fc"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "28a7ee18d29277a090275efb47ed6bf7"
server_key_iv: "6663d44d9dabfa7d214e4dfa718c97ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.684826 19541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "b0a29f44e4a8421ea78ed2fa249c71fc"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "28a7ee18d29277a090275efb47ed6bf7"
server_key_iv: "6663d44d9dabfa7d214e4dfa718c97ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.689284 19541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:11.692417 19555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:11.694162 19541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:11.694367 19541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "b0a29f44e4a8421ea78ed2fa249c71fc"
format_stamp: "Formatted at 2026-09-28 14:46:11 on dist-test-slave-gkhx"
server_key: "28a7ee18d29277a090275efb47ed6bf7"
server_key_iv: "6663d44d9dabfa7d214e4dfa718c97ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:11.694521 19541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:11.712992 19541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:11.713749 19541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:11.713972 19541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:11.721591 19541 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34897
I20260928 14:46:11.721657 19608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34897 every 8 connection(s)
I20260928 14:46:11.722831 19541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:11.726289 19609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:11.729561 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19541
I20260928 14:46:11.729696 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:11.730050 14205 external_mini_cluster.cc:1428] Setting key 028dc432f8b85d8aba0d74d16dc741dd
I20260928 14:46:11.733213 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc: Bootstrap starting.
I20260928 14:46:11.736038 19609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:11.736975 19609 log.cc:824] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:11.739289 19609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc: No bootstrap required, opened a new log
I20260928 14:46:11.741120 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:11.731622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49952 (local address 127.13.223.126:34897)
0928 14:46:11.732342 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:11.732363 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:11.732406 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:46:11.733050 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:11.733068 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:11.733544 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:11.733853 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:11.734220 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.735342 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.736648 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.737330 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.737556 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.738762 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:11.738768 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:11.738785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:11.739259 (+ 474us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:11.739326 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:11.739349 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:11.739484 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:11.739656 (+ 172us) server_negotiation.cc:299] Negotiation successful
0928 14:46:11.740094 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":166,"threads_started":1}
I20260928 14:46:11.744158 19609 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } }
I20260928 14:46:11.744526 19609 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:11.744601 19609 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a29f44e4a8421ea78ed2fa249c71fc, State: Initialized, Role: FOLLOWER
I20260928 14:46:11.744964 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:46:11.745204 19609 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } }
I20260928 14:46:11.745390 19609 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:11.745471 19609 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:11.745628 19609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:11.747074 19609 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } }
I20260928 14:46:11.747509 19609 leader_election.cc:302] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0a29f44e4a8421ea78ed2fa249c71fc; no voters:
I20260928 14:46:11.747820 19609 leader_election.cc:288] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:11.747994 19614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:11.748277 19614 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [term 1 LEADER]: Becoming Leader. State: Replica: b0a29f44e4a8421ea78ed2fa249c71fc, State: Running, Role: LEADER
I20260928 14:46:11.748651 19614 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } }
I20260928 14:46:11.749297 19609 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:11.750604 19615 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } } }
I20260928 14:46:11.750766 19615 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.751185 19623 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:11.751209 19616 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0a29f44e4a8421ea78ed2fa249c71fc. Latest consensus state: current_term: 1 leader_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0a29f44e4a8421ea78ed2fa249c71fc" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34897 } } }
I20260928 14:46:11.751377 19616 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:11.754944 19623 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:11.761749 19623 catalog_manager.cc:1347] Generated new cluster ID: a4ee6328316e471eb69149dcb934551a
I20260928 14:46:11.761847 19623 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:11.773614 19623 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:11.775170 19623 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:11.781484 19623 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0a29f44e4a8421ea78ed2fa249c71fc: Generated new TSK 0
I20260928 14:46:11.782472 19623 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:11.803051 19612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:11.796604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49954 (local address 127.13.223.126:34897)
0928 14:46:11.796767 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:11.796772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:11.796922 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:46:11.797082 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:11.797086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:11.797155 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:11.797290 (+ 135us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:46:11.797607 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.798644 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.799839 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.800278 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.800403 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.800766 (+ 363us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:46:11.802583 (+ 1817us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:46:11.802869 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 14:46:11.803140 19612 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49954: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260928 14:46:11.804919 19638 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:11.796468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34897 (local address 127.0.0.1:49954)
0928 14:46:11.796827 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:11.796840 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:11.796949 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:11.797325 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:11.797328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:11.797336 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:46:11.797480 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.797486 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.798825 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.798829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.799699 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:11.799712 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:11.800438 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:11.800441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:11.800521 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:11.800554 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:46:11.802794 (+ 2240us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 14:46:11.804728 (+ 1934us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.126:34897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43865/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":1673,"thread_start_us":116,"threads_started":1}
I20260928 14:46:11.805852 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19541
E20260928 14:46:11.817601 19639 webserver.cc:533] Webserver: sslize failed:
E20260928 14:46:11.817700 19639 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T14:46:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (338 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T14:46:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:11Z Disabled control of system clock
I20260928 14:46:11.857741 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34297
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:34413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34297
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/testchainca.cert with env {}
W20260928 14:46:11.988773 19643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:11.989117 19643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:11.989167 19643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:11.993475 19643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:11.993566 19643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:11.993582 19643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:11.993605 19643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:11.993621 19643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 14:46:11.993638 19643 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 14:46:11.993652 19643 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 14:46:11.999631 19643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34297
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:12.000836 19643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:12.001917 19643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:12.011520 19648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:12.011603 19651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:12.011682 19643 server_base.cc:1034] running on GCE node
W20260928 14:46:12.011520 19649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:12.012547 19643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:12.013849 19643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:12.015069 19643 hybrid_clock.cc:648] HybridClock initialized: now 1790606772015023 us; error 70 us; skew 500 ppm
I20260928 14:46:12.017556 19643 webserver.cc:460] Webserver started at http://127.13.223.126:44809/ using document root <none> and password file <none>
I20260928 14:46:12.018275 19643 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:12.018390 19643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:12.018736 19643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:12.021587 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "7f757219079342268511f91dd134c24e"
format_stamp: "Formatted at 2026-09-28 14:46:12 on dist-test-slave-gkhx"
server_key: "675b2566917190bf5c4627d68f2acaa1"
server_key_iv: "ace9cd9a2260148a9202f23316194217"
server_key_version: "encryptionkey@0"
I20260928 14:46:12.022362 19643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "7f757219079342268511f91dd134c24e"
format_stamp: "Formatted at 2026-09-28 14:46:12 on dist-test-slave-gkhx"
server_key: "675b2566917190bf5c4627d68f2acaa1"
server_key_iv: "ace9cd9a2260148a9202f23316194217"
server_key_version: "encryptionkey@0"
I20260928 14:46:12.027442 19643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260928 14:46:12.030537 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:12.032356 19643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:12.032527 19643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "7f757219079342268511f91dd134c24e"
format_stamp: "Formatted at 2026-09-28 14:46:12 on dist-test-slave-gkhx"
server_key: "675b2566917190bf5c4627d68f2acaa1"
server_key_iv: "ace9cd9a2260148a9202f23316194217"
server_key_version: "encryptionkey@0"
I20260928 14:46:12.032651 19643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:12.044494 19643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:12.045249 19643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:12.045451 19643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:12.053740 19643 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34297
I20260928 14:46:12.053764 19710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34297 every 8 connection(s)
I20260928 14:46:12.054983 19643 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:12.055686 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19643
I20260928 14:46:12.055805 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:12.056162 14205 external_mini_cluster.cc:1428] Setting key 4d710f4cbb5bba95766c0dfca500e08b
I20260928 14:46:12.058910 19711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:12.065979 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e: Bootstrap starting.
I20260928 14:46:12.067675 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:12.057768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49484 (local address 127.13.223.126:34297)
0928 14:46:12.058660 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:12.058687 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:12.058731 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:46:12.059590 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:12.059611 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:12.060132 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:12.060559 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:12.061082 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.062301 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.063528 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.064121 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.064335 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:12.065437 (+ 1102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:12.065444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:12.065466 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:12.065930 (+ 464us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:12.066014 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:12.066038 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:12.066181 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:12.066393 (+ 212us) server_negotiation.cc:299] Negotiation successful
0928 14:46:12.066748 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":179,"threads_started":1}
I20260928 14:46:12.068784 19711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:12.069615 19711 log.cc:824] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:12.072108 19711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e: No bootstrap required, opened a new log
I20260928 14:46:12.072306 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:46:12.076344 19711 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } }
I20260928 14:46:12.076645 19711 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:12.076728 19711 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f757219079342268511f91dd134c24e, State: Initialized, Role: FOLLOWER
I20260928 14:46:12.077288 19711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } }
I20260928 14:46:12.077436 19711 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:12.077592 19711 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:12.077787 19711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:12.079226 19711 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } }
I20260928 14:46:12.079743 19711 leader_election.cc:302] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f757219079342268511f91dd134c24e; no voters:
I20260928 14:46:12.080245 19711 leader_election.cc:288] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:12.080359 19716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:12.080600 19716 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [term 1 LEADER]: Becoming Leader. State: Replica: 7f757219079342268511f91dd134c24e, State: Running, Role: LEADER
I20260928 14:46:12.080983 19716 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } }
I20260928 14:46:12.081934 19711 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:12.083130 19718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f757219079342268511f91dd134c24e. Latest consensus state: current_term: 1 leader_uuid: "7f757219079342268511f91dd134c24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } } }
I20260928 14:46:12.083375 19717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f757219079342268511f91dd134c24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f757219079342268511f91dd134c24e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34297 } } }
I20260928 14:46:12.083519 19717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:12.083559 19718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:12.083988 19725 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:12.087999 19725 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:12.094720 19725 catalog_manager.cc:1347] Generated new cluster ID: be278eef6aa0463abfac058a5f986395
I20260928 14:46:12.094820 19725 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:12.106647 19725 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:12.107892 19725 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:12.114096 19725 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f757219079342268511f91dd134c24e: Generated new TSK 0
I20260928 14:46:12.115121 19725 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:12.220613 19714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:12.127357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.13.223.126:34297)
0928 14:46:12.127542 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:12.127547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:12.127802 (+ 255us) server_negotiation.cc:407] Connection header received
0928 14:46:12.127953 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:12.127956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:12.128012 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:12.128117 (+ 105us) server_negotiation.cc:226] Negotiated authn=JWT
0928 14:46:12.128443 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.129163 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.130856 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.131237 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.131351 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:12.131816 (+ 465us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 14:46:12.220152 (+ 88336us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:46:12.220470 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49498: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 14:46:12.220737 19714 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49498: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260928 14:46:12.221217 19740 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:12.127277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34297 (local address 127.0.0.1:49498)
0928 14:46:12.127650 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:12.127667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:12.127807 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:12.128144 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:12.128147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:12.128155 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 14:46:12.128294 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:12.128301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.129306 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.129311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:12.130334 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:12.130349 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:12.131405 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:12.131410 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:12.131530 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:12.131569 (+ 39us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 14:46:12.220457 (+ 88888us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 14:46:12.220689 (+ 232us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.13.223.126:34297: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
I20260928 14:46:12.221925 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19643
2026-09-28T14:46:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (394 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (142 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (138 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:12 dist-test-slave-gkhx krb5kdc[19753](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:12 dist-test-slave-gkhx krb5kdc[19753](info): set up 2 sockets
Sep 28 14:46:12 dist-test-slave-gkhx krb5kdc[19753](info): commencing operation
krb5kdc: starting...
W20260928 14:46:13.605789 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:13 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:13.747854 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34271
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34271
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:13.874145 19766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:13.874521 19766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:13.874637 19766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:13.879379 19766 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:13.879536 19766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:13.879582 19766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:13.879619 19766 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:13.879652 19766 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:13.885540 19766 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34271
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19766
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:13.886822 19766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:13.887988 19766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:13.897804 19774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:13.897969 19771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:13.897804 19772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:13.897962 19766 server_base.cc:1034] running on GCE node
I20260928 14:46:13.898828 19766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:13.900189 19766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:13.901407 19766 hybrid_clock.cc:648] HybridClock initialized: now 1790606773901370 us; error 53 us; skew 500 ppm
Sep 28 14:46:13 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606773, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:13.915642 19766 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:13.916766 19766 webserver.cc:460] Webserver started at http://127.13.223.126:38155/ using document root <none> and password file <none>
I20260928 14:46:13.917486 19766 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:13.917558 19766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:13.917801 19766 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:13.920204 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "1040d91699704222ad68edd58aa37f7e"
format_stamp: "Formatted at 2026-09-28 14:46:13 on dist-test-slave-gkhx"
server_key: "78b4db45251cae369ad7eb69fd8ba631"
server_key_iv: "bf425de7886d37a21298c6621cd9153f"
server_key_version: "encryptionkey@0"
I20260928 14:46:13.920856 19766 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "1040d91699704222ad68edd58aa37f7e"
format_stamp: "Formatted at 2026-09-28 14:46:13 on dist-test-slave-gkhx"
server_key: "78b4db45251cae369ad7eb69fd8ba631"
server_key_iv: "bf425de7886d37a21298c6621cd9153f"
server_key_version: "encryptionkey@0"
I20260928 14:46:13.925480 19766 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:13.928445 19781 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:13.930061 19766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:13.930200 19766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "1040d91699704222ad68edd58aa37f7e"
format_stamp: "Formatted at 2026-09-28 14:46:13 on dist-test-slave-gkhx"
server_key: "78b4db45251cae369ad7eb69fd8ba631"
server_key_iv: "bf425de7886d37a21298c6621cd9153f"
server_key_version: "encryptionkey@0"
I20260928 14:46:13.930393 19766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:13.956230 19766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:13.960692 19766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:13.960950 19766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:13.969571 19766 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34271
I20260928 14:46:13.969632 19833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34271 every 8 connection(s)
I20260928 14:46:13.970779 19766 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:13.974135 19834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:13.976919 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19766
I20260928 14:46:13.977073 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:13.977385 14205 external_mini_cluster.cc:1428] Setting key 529ef16f0f36841cb0fdc143d7a18c1b
I20260928 14:46:13.982263 19834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: Bootstrap starting.
I20260928 14:46:13.985500 19834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:13.986465 19834 log.cc:824] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:46:13 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606773, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:13.989346 19834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: No bootstrap required, opened a new log
I20260928 14:46:13.994385 19834 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } }
I20260928 14:46:13.994823 19834 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:13.994906 19834 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1040d91699704222ad68edd58aa37f7e, State: Initialized, Role: FOLLOWER
I20260928 14:46:13.995404 19834 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } }
I20260928 14:46:13.995538 19834 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:13.995636 19834 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:13.995770 19834 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:13.996816 19834 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } }
I20260928 14:46:13.997162 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:13.979414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55598 (local address 127.13.223.126:34271)
0928 14:46:13.979936 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:13.979959 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:13.980016 (+ 57us) server_negotiation.cc:407] Connection header received
0928 14:46:13.981126 (+ 1110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:13.981150 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:13.981576 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:13.981916 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:13.983279 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:13.984388 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:13.985805 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:13.986486 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:13.986755 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:13.990028 (+ 3273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:13.990034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:13.990051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:13.990085 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:13.992769 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:13.993465 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:13.993473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:13.993481 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:13.993633 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:13.994016 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:13.994019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:13.994021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:13.994527 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:13.994837 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:13.995583 (+ 746us) server_negotiation.cc:299] Negotiation successful
0928 14:46:13.995963 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":173,"threads_started":1}
I20260928 14:46:13.997222 19834 leader_election.cc:302] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1040d91699704222ad68edd58aa37f7e; no voters:
I20260928 14:46:13.997643 19834 leader_election.cc:288] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:13.997747 19839 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:13.997915 19839 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [term 1 LEADER]: Becoming Leader. State: Replica: 1040d91699704222ad68edd58aa37f7e, State: Running, Role: LEADER
I20260928 14:46:13.998311 19839 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } }
I20260928 14:46:13.999117 19834 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:14.000228 19840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1040d91699704222ad68edd58aa37f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } } }
I20260928 14:46:14.000376 19840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:14.000589 19841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1040d91699704222ad68edd58aa37f7e. Latest consensus state: current_term: 1 leader_uuid: "1040d91699704222ad68edd58aa37f7e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1040d91699704222ad68edd58aa37f7e" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34271 } } }
I20260928 14:46:14.000792 19841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e [sys.catalog]: This master's current role is: LEADER
W20260928 14:46:14.004665 19854 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:46:14.004803 19854 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:46:14.004873 19848 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:14.005637 19848 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:14.012014 19848 catalog_manager.cc:1347] Generated new cluster ID: f5d8084d1c8a4ff48ff7a46a7b73e4e9
I20260928 14:46:14.012125 19848 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:14.018824 19848 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:14.020331 19848 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:14.034055 19848 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1040d91699704222ad68edd58aa37f7e: Generated new TSK 0
I20260928 14:46:14.035140 19848 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:46:14.075927 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:14.207336 19860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:14.207810 19860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:14.207894 19860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:14.212735 19860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:14.212885 19860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:14.213006 19860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:14.219368 19860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19860
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:14.220633 19860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:14.221804 19860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:14.234755 19865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.235059 19868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.235160 19866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:14.235227 19860 server_base.cc:1034] running on GCE node
I20260928 14:46:14.235985 19860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:14.237047 19860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:14.238463 19860 hybrid_clock.cc:648] HybridClock initialized: now 1790606774238408 us; error 101 us; skew 500 ppm
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.241911 19860 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:14.243098 19860 webserver.cc:460] Webserver started at http://127.13.223.65:42873/ using document root <none> and password file <none>
I20260928 14:46:14.243767 19860 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:14.243873 19860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:14.244138 19860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:14.246639 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "44eaededddc94233b05434b84264dbeb"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "6051171b9baaa6adcc1149a86f36b6a3"
server_key_iv: "c5a98f9abcb9d2290811f489598ab99a"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.247401 19860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "44eaededddc94233b05434b84264dbeb"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "6051171b9baaa6adcc1149a86f36b6a3"
server_key_iv: "c5a98f9abcb9d2290811f489598ab99a"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.252169 19860 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:14.255569 19875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.257320 19860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:14.257551 19860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "44eaededddc94233b05434b84264dbeb"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "6051171b9baaa6adcc1149a86f36b6a3"
server_key_iv: "c5a98f9abcb9d2290811f489598ab99a"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.257709 19860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:14.283893 19860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:14.288481 19860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:14.288733 19860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:14.289430 19860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:14.290398 19860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:14.290477 19860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.290585 19860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:14.290647 19860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.301698 19860 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:39101
I20260928 14:46:14.301734 19989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:39101 every 8 connection(s)
I20260928 14:46:14.302896 19860 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:14.305684 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19860
I20260928 14:46:14.305797 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:14.306203 14205 external_mini_cluster.cc:1428] Setting key 4a7b3d31b1808c87e63b6382451c9c89
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:46:14.322525 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.305262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:38123 (local address 127.13.223.126:34271)
0928 14:46:14.305440 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.305446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.306796 (+ 1350us) server_negotiation.cc:407] Connection header received
0928 14:46:14.308574 (+ 1778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.308580 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.308656 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.308760 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.310961 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.311643 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.313310 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.313618 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.313764 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.317330 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.317347 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.317350 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.317389 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.319485 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.320320 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.320357 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.320364 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.320503 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.321032 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.321038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.321042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.321340 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.321456 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.322170 (+ 714us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.322377 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:46:14.324067 19992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.305915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34271 (local address 127.13.223.65:38123)
0928 14:46:14.306541 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:14.306643 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:14.308299 (+ 1656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:14.309069 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:14.309089 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:14.309691 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:14.310614 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.310647 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.311847 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.311851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.313135 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.313148 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.313940 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.313948 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.314164 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.315063 (+ 899us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:14.315083 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:14.317067 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:14.319692 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.319701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.319726 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.320117 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.320622 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.320630 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.320635 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.320892 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.321494 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:14.321501 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:14.321853 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 14:46:14.322051 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 14:46:14.322591 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":179,"threads_started":1}
I20260928 14:46:14.325634 19990 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34271
I20260928 14:46:14.325965 19990 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:14.326685 19990 heartbeater.cc:511] Master 127.13.223.126:34271 requested a full tablet report, sending...
I20260928 14:46:14.328583 19792 ts_manager.cc:194] Registered new tserver with Master: 44eaededddc94233b05434b84264dbeb (127.13.223.65:39101)
I20260928 14:46:14.329998 19792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:38123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:14.344739 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:14.473719 19995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:14.474100 19995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:14.474186 19995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:14.478513 19995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:14.478683 19995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:14.478789 19995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:14.484896 19995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:14.486199 19995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:14.487459 19995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:14.498765 20003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.498756 20000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.498756 20001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:14.499176 19995 server_base.cc:1034] running on GCE node
I20260928 14:46:14.499811 19995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:14.500769 19995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:14.501991 19995 hybrid_clock.cc:648] HybridClock initialized: now 1790606774501945 us; error 77 us; skew 500 ppm
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.505635 19995 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:14.507175 19995 webserver.cc:460] Webserver started at http://127.13.223.66:37947/ using document root <none> and password file <none>
I20260928 14:46:14.507830 19995 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:14.507946 19995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:14.508176 19995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:14.510723 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "07a6d81faedc462eab268625faed937b"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "86179dc4d86dc26d4eda5dd86cdf4d9e"
server_key_iv: "e0b3dad7701629ad555fd0ce7e82afaa"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.511682 19995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "07a6d81faedc462eab268625faed937b"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "86179dc4d86dc26d4eda5dd86cdf4d9e"
server_key_iv: "e0b3dad7701629ad555fd0ce7e82afaa"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.516696 19995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:46:14.520001 20010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.521835 19995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:14.522076 19995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "07a6d81faedc462eab268625faed937b"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "86179dc4d86dc26d4eda5dd86cdf4d9e"
server_key_iv: "e0b3dad7701629ad555fd0ce7e82afaa"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.522280 19995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:14.544138 19995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:14.548272 19995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:14.548547 19995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:14.549268 19995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:14.550289 19995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:14.550371 19995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.550453 19995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:14.550522 19995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.561698 19995 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:46459
I20260928 14:46:14.561720 20124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:46459 every 8 connection(s)
I20260928 14:46:14.562947 19995 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:14.563277 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 19995
I20260928 14:46:14.563406 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:14.563791 14205 external_mini_cluster.cc:1428] Setting key ac3db7eef247e84764f077f246f567b4
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:14.583647 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.565063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:40277 (local address 127.13.223.126:34271)
0928 14:46:14.565219 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.565224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.566396 (+ 1172us) server_negotiation.cc:407] Connection header received
0928 14:46:14.568021 (+ 1625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.568024 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.568090 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.568210 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.569563 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.570167 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.571622 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.571885 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.572020 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.578345 (+ 6325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.578350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.578352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.578388 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.580509 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.581468 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.581472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.581473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.581590 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.582148 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.582152 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.582153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.582397 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.582510 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.583373 (+ 863us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.583510 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:46:14.585122 20127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.565601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34271 (local address 127.13.223.66:40277)
0928 14:46:14.566231 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:14.566279 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:14.567751 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:14.568411 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:14.568420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:14.568834 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:14.569386 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.569409 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.570322 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.570326 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.571459 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.571472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.572075 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.572080 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.572276 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.573030 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:14.573049 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:14.578037 (+ 4988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:14.580705 (+ 2668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.580717 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.580742 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.581300 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.581702 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.581708 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.581713 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.581964 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.582544 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:14.582574 (+ 30us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:14.583026 (+ 452us) client_negotiation.cc:769] Sending connection context
0928 14:46:14.583290 (+ 264us) client_negotiation.cc:240] Negotiation successful
0928 14:46:14.583758 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":151,"threads_started":1}
I20260928 14:46:14.586759 20125 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34271
I20260928 14:46:14.587141 20125 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:14.587738 20125 heartbeater.cc:511] Master 127.13.223.126:34271 requested a full tablet report, sending...
I20260928 14:46:14.589093 19792 ts_manager.cc:194] Registered new tserver with Master: 07a6d81faedc462eab268625faed937b (127.13.223.66:46459)
I20260928 14:46:14.589799 19792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:40277
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:14.599299 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:14.719291 20130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:14.719638 20130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:14.719725 20130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:14.724157 20130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:14.724268 20130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:14.724364 20130 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:14.730404 20130 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:34271
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:14.731669 20130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:14.732780 20130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:14.745319 20138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.745333 20136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:14.745360 20135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:14.745661 20130 server_base.cc:1034] running on GCE node
I20260928 14:46:14.746268 20130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:14.747162 20130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:14.748412 20130 hybrid_clock.cc:648] HybridClock initialized: now 1790606774748391 us; error 37 us; skew 500 ppm
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.752104 20130 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:14.753219 20130 webserver.cc:460] Webserver started at http://127.13.223.67:35689/ using document root <none> and password file <none>
I20260928 14:46:14.753882 20130 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:14.753994 20130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:14.754251 20130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:14.756618 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "5c7859ff30a447889f76c053d88379f7"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "5f6fcf9e816816172bc1ab09425e1066"
server_key_iv: "33fa532713147bdf1b89f8860066c985"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.757290 20130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "5c7859ff30a447889f76c053d88379f7"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "5f6fcf9e816816172bc1ab09425e1066"
server_key_iv: "33fa532713147bdf1b89f8860066c985"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.762609 20130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:46:14.765758 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.767643 20130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:14.767824 20130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "5c7859ff30a447889f76c053d88379f7"
format_stamp: "Formatted at 2026-09-28 14:46:14 on dist-test-slave-gkhx"
server_key: "5f6fcf9e816816172bc1ab09425e1066"
server_key_iv: "33fa532713147bdf1b89f8860066c985"
server_key_version: "encryptionkey@0"
I20260928 14:46:14.767982 20130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:14.781970 20130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:14.785979 20130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:14.786242 20130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:14.787063 20130 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:14.788012 20130 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:14.788062 20130 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.788105 20130 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:14.788164 20130 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:14.797839 20130 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:43959
I20260928 14:46:14.797876 20259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:43959 every 8 connection(s)
I20260928 14:46:14.798941 20130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:14.807534 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20130
I20260928 14:46:14.807719 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:14.808085 14205 external_mini_cluster.cc:1428] Setting key 7545e5b4ab423c3d01eb812368743a4c
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:14.816092 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.801206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44915 (local address 127.13.223.126:34271)
0928 14:46:14.801407 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.801411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.802456 (+ 1045us) server_negotiation.cc:407] Connection header received
0928 14:46:14.803985 (+ 1529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.803989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.804058 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.804174 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.805561 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.806128 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.807498 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.807821 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.807959 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.811458 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.811462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.811464 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.811492 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.813124 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.813771 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.813775 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.813776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.813947 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.814461 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.814464 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.814465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.814768 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.814938 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.815804 (+ 866us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.815933 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260928 14:46:14.817016 20262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.801713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:34271 (local address 127.13.223.67:44915)
0928 14:46:14.802260 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:14.802306 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:14.803708 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:14.804381 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:14.804389 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:14.804813 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:14.805356 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.805370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.806281 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.806287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.807314 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:14.807342 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.808006 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.808011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:14.808252 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.809066 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:14.809094 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:14.811255 (+ 2161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:14.813265 (+ 2010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.813271 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.813289 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.813634 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.814088 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:14.814093 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:14.814095 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:14.814345 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:14.815130 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:14.815136 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:14.815522 (+ 386us) client_negotiation.cc:769] Sending connection context
0928 14:46:14.815708 (+ 186us) client_negotiation.cc:240] Negotiation successful
0928 14:46:14.816032 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":133,"threads_started":1}
I20260928 14:46:14.818391 20260 heartbeater.cc:347] Connected to a master server at 127.13.223.126:34271
I20260928 14:46:14.818735 20260 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:14.819536 20260 heartbeater.cc:511] Master 127.13.223.126:34271 requested a full tablet report, sending...
I20260928 14:46:14.820889 19792 ts_manager.cc:194] Registered new tserver with Master: 5c7859ff30a447889f76c053d88379f7 (127.13.223.67:43959)
I20260928 14:46:14.821693 19792 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:44915
I20260928 14:46:14.823786 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/test-admin.keytab.
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.845386 14205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 14:46:14.849432 20267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:14.847306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47570 (local address 127.13.223.65:39101)
0928 14:46:14.847728 (+ 422us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.847739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.847777 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:46:14.847988 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.847996 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.848296 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.848487 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.849220 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47570: BlockingRecv error: recv got EOF from 127.0.0.1:47570 (error 108)
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":118,"threads_started":1}
W20260928 14:46:14.850026 14205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.851413 14205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.852794 14205 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:46:14.867792 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.854633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.13.223.65:39101)
0928 14:46:14.854793 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.854800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.854903 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:46:14.855076 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.855079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.855135 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.855231 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.855934 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.856764 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.858109 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.858840 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.858995 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.862388 (+ 3393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.862407 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.862423 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.862482 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.865103 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.865726 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.865733 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.865734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.865933 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.866428 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.866432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.866433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.866895 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.867120 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.867491 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.867654 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260928 14:46:14.872519 20267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:14.871245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.13.223.65:39101)
0928 14:46:14.871407 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.871412 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.871530 (+ 118us) server_negotiation.cc:407] Connection header received
0928 14:46:14.871704 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.871707 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.871762 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.871866 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.872366 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47584: BlockingRecv error: recv got EOF from 127.0.0.1:47584 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 14:46:14.873008 14205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.874171 14205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.875162 14205 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:46:14.886124 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.876128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.13.223.65:39101)
0928 14:46:14.876364 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.876370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.876477 (+ 107us) server_negotiation.cc:407] Connection header received
0928 14:46:14.876653 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.876657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.876711 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.876805 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.877448 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.878066 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.879038 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.879299 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.879412 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.882100 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.882103 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.882105 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.882137 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.884038 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.884575 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.884578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.884579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.884666 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.885054 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.885057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.885059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.885264 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.885411 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.885701 (+ 290us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.885944 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 14:46:14.889884 20267 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:14.888520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47608 (local address 127.13.223.65:39101)
0928 14:46:14.888693 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.888696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.888711 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:46:14.888817 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.888820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.888863 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.888939 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.889743 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47608: BlockingRecv error: recv got EOF from 127.0.0.1:47608 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 14:46:14.889909 14205 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790606739990168-14205-0/krb5kdc/krb5cc' of type FILE
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.891153 14205 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:14.892123 14205 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[19753](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606774, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:46:14.903574 20267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:14.893242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47612 (local address 127.13.223.65:39101)
0928 14:46:14.893436 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:14.893442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:14.893602 (+ 160us) server_negotiation.cc:407] Connection header received
0928 14:46:14.893782 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:14.893786 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:14.893844 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:14.894080 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:14.894828 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.895415 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.896394 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:14.896747 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:14.896884 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:14.899654 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:14.899658 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:14.899660 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:14.899691 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:14.901579 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.902187 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.902191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.902193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.902287 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:14.902632 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:14.902636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:14.902638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:14.902847 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:14.903036 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:14.903280 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 14:46:14.903396 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 14:46:14.904989 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19860
I20260928 14:46:14.911796 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19995
I20260928 14:46:14.917744 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20130
I20260928 14:46:14.924193 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 19766
2026-09-28T14:46:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2423 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[20280](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[20280](info): set up 2 sockets
Sep 28 14:46:14 dist-test-slave-gkhx krb5kdc[20280](info): commencing operation
krb5kdc: starting...
W20260928 14:46:16.053308 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.105s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab.
Entry for principal oryx/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab.
I20260928 14:46:16.200145 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44095
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab
--principal=oryx/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:16.326339 20293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:16.326804 20293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:16.326859 20293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:16.331310 20293 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:16.331400 20293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:16.331423 20293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:16.331444 20293 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:16.331463 20293 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:16.338132 20293 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab
--principal=oryx/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:16.339438 20293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:16.340605 20293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:16.351024 20301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:16.351024 20298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:16.351060 20299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:16.351181 20293 server_base.cc:1034] running on GCE node
I20260928 14:46:16.352138 20293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:16.353433 20293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:16.354684 20293 hybrid_clock.cc:648] HybridClock initialized: now 1790606776354688 us; error 107 us; skew 500 ppm
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:16.358034 20293 init.cc:377] Logged in from keytab as oryx/127.13.223.126@KRBTEST.COM (short username oryx)
I20260928 14:46:16.359189 20293 webserver.cc:460] Webserver started at http://127.13.223.126:44517/ using document root <none> and password file <none>
I20260928 14:46:16.359827 20293 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:16.359928 20293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:16.360183 20293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:16.362493 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "4f8556b66b28e20dd5e77fd9561b391c"
server_key_iv: "a0580d0fa57ac523bb4f36b0530fa5a0"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.363199 20293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "4f8556b66b28e20dd5e77fd9561b391c"
server_key_iv: "a0580d0fa57ac523bb4f36b0530fa5a0"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.367599 20293 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:46:16.370528 20308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.372156 20293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:16.372391 20293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "4f8556b66b28e20dd5e77fd9561b391c"
server_key_iv: "a0580d0fa57ac523bb4f36b0530fa5a0"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.372553 20293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:16.392664 20293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:16.396576 20293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:16.396821 20293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:16.404901 20293 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44095
I20260928 14:46:16.404971 20360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44095 every 8 connection(s)
I20260928 14:46:16.406030 20293 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:16.408545 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20293
I20260928 14:46:16.408664 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:16.409014 14205 external_mini_cluster.cc:1428] Setting key 65af7c9c4102c827ffcd55f37c311336
I20260928 14:46:16.409441 20361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:16.416610 20361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Bootstrap starting.
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:16.420092 20361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:16.421129 20361 log.cc:824] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:16.423823 20361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: No bootstrap required, opened a new log
I20260928 14:46:16.426280 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:16.410660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42506 (local address 127.13.223.126:44095)
0928 14:46:16.411175 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:16.411192 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:16.411229 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:46:16.411923 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:16.411941 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:16.412329 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:16.412736 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:16.413752 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.415035 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.416006 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.416557 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.416758 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:16.419869 (+ 3111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:16.419878 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:16.419901 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:16.419963 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:16.422404 (+ 2441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:16.423194 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:16.423201 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:16.423207 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:16.423365 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:16.423764 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:16.423768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:16.423769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:16.424356 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:16.424627 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:16.425027 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:46:16.425406 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":159,"threads_started":1}
I20260928 14:46:16.428179 20361 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } }
I20260928 14:46:16.428474 20361 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:16.428565 20361 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda5c6fa51af4e8c81f35f3b6f1e8828, State: Initialized, Role: FOLLOWER
I20260928 14:46:16.429085 20361 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } }
I20260928 14:46:16.429256 20361 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:16.429348 20361 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:16.429482 20361 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:16.430676 20361 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } }
I20260928 14:46:16.431059 20361 leader_election.cc:302] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda5c6fa51af4e8c81f35f3b6f1e8828; no voters:
I20260928 14:46:16.431425 20361 leader_election.cc:288] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:16.431644 20366 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:16.432016 20366 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 LEADER]: Becoming Leader. State: Replica: cda5c6fa51af4e8c81f35f3b6f1e8828, State: Running, Role: LEADER
I20260928 14:46:16.432407 20366 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } }
I20260928 14:46:16.432909 20361 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:16.434525 20368 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5c6fa51af4e8c81f35f3b6f1e8828. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } }
I20260928 14:46:16.434589 20367 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } }
I20260928 14:46:16.434708 20368 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:16.434708 20367 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:16.435145 20376 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:16.438835 20376 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:16.444819 20376 catalog_manager.cc:1347] Generated new cluster ID: dd07be5537b5430daf44285a88ecf056
I20260928 14:46:16.444908 20376 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:16.466264 20376 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:16.467730 20376 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:16.483880 20376 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Generated new TSK 0
I20260928 14:46:16.484835 20376 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
I20260928 14:46:16.519418 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:16.646983 20387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:16.647290 20387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:16.647343 20387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:16.651678 20387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:16.651757 20387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:16.651820 20387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:16.657560 20387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:16.658799 20387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:16.659888 20387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:16.670823 20395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:16.671201 20387 server_base.cc:1034] running on GCE node
W20260928 14:46:16.670855 20393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:16.670826 20392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:16.671869 20387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:16.672696 20387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:16.673924 20387 hybrid_clock.cc:648] HybridClock initialized: now 1790606776673903 us; error 38 us; skew 500 ppm
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:16.677459 20387 init.cc:377] Logged in from keytab as oryx/127.13.223.65@KRBTEST.COM (short username oryx)
I20260928 14:46:16.678537 20387 webserver.cc:460] Webserver started at http://127.13.223.65:44287/ using document root <none> and password file <none>
I20260928 14:46:16.679200 20387 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:16.679265 20387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:16.679447 20387 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:16.681597 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "0c83b31373ec423c8f4524befc90bfc4"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "d4d352b1a68afe5fba7102ea0f073b0c"
server_key_iv: "3cf42e9bd76ce155a17416f27608f286"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.682153 20387 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "0c83b31373ec423c8f4524befc90bfc4"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "d4d352b1a68afe5fba7102ea0f073b0c"
server_key_iv: "3cf42e9bd76ce155a17416f27608f286"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.686610 20387 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.004s
I20260928 14:46:16.689338 20402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.691074 20387 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:16.691211 20387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "0c83b31373ec423c8f4524befc90bfc4"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "d4d352b1a68afe5fba7102ea0f073b0c"
server_key_iv: "3cf42e9bd76ce155a17416f27608f286"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.691330 20387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:16.714702 20387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:16.718700 20387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:16.718905 20387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:16.719520 20387 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:16.720404 20387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:16.720460 20387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.720506 20387 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:16.720525 20387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.731261 20387 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:41187
I20260928 14:46:16.731285 20515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:41187 every 8 connection(s)
I20260928 14:46:16.732366 20387 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:16.737812 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20387
I20260928 14:46:16.737953 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:16.738288 14205 external_mini_cluster.cc:1428] Setting key fef9789b8ca0d475905b28c0252d1126
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:46:16.750952 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:16.734696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:38639 (local address 127.13.223.126:44095)
0928 14:46:16.734855 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:16.734860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:16.736010 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 14:46:16.737407 (+ 1397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:16.737410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:16.737471 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:16.737553 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:16.739358 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.740270 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.742288 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.742777 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.742924 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:16.746002 (+ 3078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:16.746010 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:16.746017 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:16.746060 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:16.748189 (+ 2129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:16.748844 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:16.748850 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:16.748854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:16.748958 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:16.749417 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:16.749421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:16.749423 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:16.749731 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:16.749871 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:16.750491 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 14:46:16.750705 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:16.751981 20518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:16.735275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:38639)
0928 14:46:16.735816 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:16.735864 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:16.737170 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:16.737816 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:16.737826 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:16.738385 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:16.739162 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:16.739185 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.740433 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.740439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:16.742102 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:16.742115 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:16.742990 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:16.742993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:16.743159 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:16.743834 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:16.743852 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:16.745764 (+ 1912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:16.748370 (+ 2606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:16.748377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:16.748393 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:16.748718 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:16.749122 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:16.749125 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:16.749126 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:16.749299 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:16.749881 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:16.749887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:16.750199 (+ 312us) client_negotiation.cc:769] Sending connection context
0928 14:46:16.750408 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 14:46:16.750846 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":132,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.66@KRBTEST.COM" created.
I20260928 14:46:16.753793 20516 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:16.754168 20516 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:16.754845 20516 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:16.756691 20325 ts_manager.cc:194] Registered new tserver with Master: 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187)
I20260928 14:46:16.758126 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:38639
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
I20260928 14:46:16.775182 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:16.897131 20521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:16.897457 20521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:16.897503 20521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:16.901818 20521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:16.901887 20521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:16.901950 20521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:16.907595 20521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:16.908804 20521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:16.909855 20521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:16.921316 20527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:16.921411 20521 server_base.cc:1034] running on GCE node
W20260928 14:46:16.921330 20529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:16.921322 20526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:16.922336 20521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:16.923206 20521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:16.924400 20521 hybrid_clock.cc:648] HybridClock initialized: now 1790606776924379 us; error 36 us; skew 500 ppm
Sep 28 14:46:16 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:16.928012 20521 init.cc:377] Logged in from keytab as oryx/127.13.223.66@KRBTEST.COM (short username oryx)
I20260928 14:46:16.929117 20521 webserver.cc:460] Webserver started at http://127.13.223.66:33475/ using document root <none> and password file <none>
I20260928 14:46:16.929769 20521 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:16.929885 20521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:16.930159 20521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:16.932585 20521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "504a5636894b4f94bbae983fd2717e57"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "1175a98399f9ff93f17ee9f21906ff70"
server_key_iv: "a78c90af015b47f57708b94a07d26512"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.933323 20521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "504a5636894b4f94bbae983fd2717e57"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "1175a98399f9ff93f17ee9f21906ff70"
server_key_iv: "a78c90af015b47f57708b94a07d26512"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.937985 20521 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:46:16.940965 20536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.942447 20521 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:16.942672 20521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "504a5636894b4f94bbae983fd2717e57"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "1175a98399f9ff93f17ee9f21906ff70"
server_key_iv: "a78c90af015b47f57708b94a07d26512"
server_key_version: "encryptionkey@0"
I20260928 14:46:16.942819 20521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:16.980892 20521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:16.985149 20521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:16.985426 20521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:16.986114 20521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:16.987068 20521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:16.987150 20521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.987231 20521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:16.987277 20521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:16.998081 20521 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:41435
I20260928 14:46:16.998108 20650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:41435 every 8 connection(s)
I20260928 14:46:16.999189 20521 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:17.003798 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20521
I20260928 14:46:17.003944 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:17.004331 14205 external_mini_cluster.cc:1428] Setting key 3b5f83a9b3d3d5b9db54c3d8332cd55a
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for oryx/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.67@KRBTEST.COM" created.
I20260928 14:46:17.017800 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.001561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:60373 (local address 127.13.223.126:44095)
0928 14:46:17.001720 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.001724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.002629 (+ 905us) server_negotiation.cc:407] Connection header received
0928 14:46:17.004613 (+ 1984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.004617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.004672 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.004781 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.006504 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.007143 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.008844 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.009283 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.009411 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.012873 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.012877 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.012881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.012915 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.014973 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.015850 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.015854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.015856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.015951 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.016369 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.016372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.016374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.016602 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.016765 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.017465 (+ 700us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.017620 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:46:17.018963 20653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.001932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:60373)
0928 14:46:17.002425 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.002473 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.004303 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.005005 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.005013 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.005572 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.006329 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.006353 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.007293 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.007300 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.008683 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.008702 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.009464 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.009468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.009674 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.010519 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.010572 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.012525 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.015136 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.015142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.015160 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.015680 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.016067 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.016069 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.016071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.016243 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.016784 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.016790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.017089 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.017279 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.017717 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":123,"threads_started":1}
I20260928 14:46:17.020429 20651 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:17.020789 20651 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:17.021672 20651 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:17.023340 20325 ts_manager.cc:194] Registered new tserver with Master: 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
I20260928 14:46:17.024106 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:60373
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
I20260928 14:46:17.041054 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:17.171852 20656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:17.172228 20656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:17.172310 20656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:17.176752 20656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:17.176867 20656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:17.176980 20656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:17.183804 20656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:17.185171 20656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:17.186430 20656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:17.198518 20664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.198498 20661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.198527 20662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.198916 20656 server_base.cc:1034] running on GCE node
I20260928 14:46:17.199378 20656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:17.200054 20656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:17.201290 20656 hybrid_clock.cc:648] HybridClock initialized: now 1790606777201272 us; error 31 us; skew 500 ppm
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:17.204610 20656 init.cc:377] Logged in from keytab as oryx/127.13.223.67@KRBTEST.COM (short username oryx)
I20260928 14:46:17.205687 20656 webserver.cc:460] Webserver started at http://127.13.223.67:41251/ using document root <none> and password file <none>
I20260928 14:46:17.206333 20656 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:17.206451 20656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:17.206774 20656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:17.209241 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "4b8612fb089f409d9faffefe7c9cdec4"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "dc9815be7099cc8f724fd402222ceb13"
server_key_iv: "45a74954d4e0073a447e50c33ded7abe"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.209908 20656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "4b8612fb089f409d9faffefe7c9cdec4"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "dc9815be7099cc8f724fd402222ceb13"
server_key_iv: "45a74954d4e0073a447e50c33ded7abe"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.214469 20656 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 14:46:17.217588 20671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.219472 20656 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:46:17.219678 20656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "4b8612fb089f409d9faffefe7c9cdec4"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "dc9815be7099cc8f724fd402222ceb13"
server_key_iv: "45a74954d4e0073a447e50c33ded7abe"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.219828 20656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:17.236519 20656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:17.240692 20656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:17.241016 20656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:17.241811 20656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:17.242898 20656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:17.242980 20656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.243062 20656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:17.243122 20656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.254815 20656 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:45707
I20260928 14:46:17.254966 20784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:45707 every 8 connection(s)
I20260928 14:46:17.256017 20656 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:17.259621 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20656
I20260928 14:46:17.259744 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:17.260063 14205 external_mini_cluster.cc:1428] Setting key f6b23f945ab3e6a55865fe280806c139
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:17.272616 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.258402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:59083 (local address 127.13.223.126:44095)
0928 14:46:17.258611 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.258616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.259698 (+ 1082us) server_negotiation.cc:407] Connection header received
0928 14:46:17.261473 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.261476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.261566 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.261646 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.263088 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.263665 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.264969 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.265219 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.265414 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.268407 (+ 2993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.268411 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.268413 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.268443 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.270046 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.270777 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.270780 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.270781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.270872 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.271300 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.271303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.271305 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.271534 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.271658 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.272275 (+ 617us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.272424 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 14:46:17.273772 20787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.258873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:59083)
0928 14:46:17.259466 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.259519 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.261233 (+ 1714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.261848 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.261857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.262279 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.262904 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.262920 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.263825 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.263829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.264812 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.264823 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.265428 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.265432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.265674 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.266448 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.266479 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.268168 (+ 1689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.270215 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.270223 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.270245 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.270634 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.270982 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.270985 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.270987 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.271183 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.271657 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.271666 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.271987 (+ 321us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.272210 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.272603 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":135,"threads_started":1}
I20260928 14:46:17.275516 20785 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:17.275823 20785 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:17.276417 20785 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:17.278057 20325 ts_manager.cc:194] Registered new tserver with Master: 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:17.278784 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:59083
I20260928 14:46:17.288503 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.223.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.125.keytab.
Entry for principal oryx/127.13.223.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.125.keytab.
I20260928 14:46:17.325771 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:43405
--webserver_interface=127.13.223.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.125.keytab
--principal=oryx/127.13.223.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44095,127.13.223.125:43405
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:17.451788 20790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:17.452162 20790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:17.452255 20790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:17.457099 20790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:17.457255 20790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:17.457307 20790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:17.457346 20790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:17.457383 20790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:17.463711 20790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44095,127.13.223.125:43405
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.125.keytab
--principal=oryx/127.13.223.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:43405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
W20260928 14:46:17.464035 20790 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.13.223.126:44095,127.13.223.125:43405'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 14:46:17.465046 20790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:17.466169 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:46:17.475870 20790 server_base.cc:1034] running on GCE node
W20260928 14:46:17.475986 20798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.476087 20796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.476054 20795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.477020 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:17.478436 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:17.479619 20790 hybrid_clock.cc:648] HybridClock initialized: now 1790606777479605 us; error 35 us; skew 500 ppm
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:17.483314 20790 init.cc:377] Logged in from keytab as oryx/127.13.223.125@KRBTEST.COM (short username oryx)
I20260928 14:46:17.484506 20790 webserver.cc:460] Webserver started at http://127.13.223.125:38231/ using document root <none> and password file <none>
I20260928 14:46:17.485249 20790 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:17.485325 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:17.485587 20790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:17.488111 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/instance:
uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "49948943d49f59631f5f1b8606be6813"
server_key_iv: "3ded0ed3a74ef44c4bd617ebec82a8b8"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.488806 20790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal/instance:
uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "49948943d49f59631f5f1b8606be6813"
server_key_iv: "3ded0ed3a74ef44c4bd617ebec82a8b8"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.494015 20790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:46:17.497308 20805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.499030 20790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:17.499209 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "49948943d49f59631f5f1b8606be6813"
server_key_iv: "3ded0ed3a74ef44c4bd617ebec82a8b8"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.499354 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:17.517285 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:17.521238 20790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:17.521570 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:17.530299 20790 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.125:43405
I20260928 14:46:17.530613 20857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.125:43405 every 8 connection(s)
I20260928 14:46:17.531601 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
I20260928 14:46:17.534386 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20790
I20260928 14:46:17.535308 20858 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:17.540072 20858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.125@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:17.552147 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.536289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51416 (local address 127.13.223.125:43405)
0928 14:46:17.536862 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.536881 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.536924 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:46:17.537676 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.537698 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.538099 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.538517 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.539478 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.540698 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.542108 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.542830 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.543014 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.545789 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.545795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.545805 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.545834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.548062 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.548749 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.548757 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.548772 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.548945 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.549290 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.549296 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.549298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.549953 (+ 655us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.550171 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.550621 (+ 450us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.550987 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":164,"threads_started":1}
I20260928 14:46:17.553052 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.542082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.13.223.126:44095)
0928 14:46:17.542274 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.542282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.542844 (+ 562us) server_negotiation.cc:407] Connection header received
0928 14:46:17.542995 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.542998 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.543042 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.543243 (+ 201us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.543946 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.544467 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.545826 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.546102 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.546227 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.549014 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.549022 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.549028 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.549061 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.550990 (+ 1929us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.551568 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.551572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.551574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.551675 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.551949 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.551952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.551954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.552267 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.552377 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.552812 (+ 435us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.552926 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:46:17.553146 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.542087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42518)
0928 14:46:17.542691 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.542744 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.542881 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.543294 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.543299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.543687 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.543815 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.543821 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.544724 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.544728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.545686 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.545697 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.546230 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.546234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.546361 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.547050 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.547063 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.548833 (+ 1770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.551206 (+ 2373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.551215 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.551220 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.551450 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.551774 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.551776 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.551777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.551855 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.552436 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.552442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.552573 (+ 131us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.552737 (+ 164us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.552951 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":196,"threads_started":1}
I20260928 14:46:17.556188 20858 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.125@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
I20260928 14:46:17.565572 20862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.556799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51424)
0928 14:46:17.556944 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.556961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.557056 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.557298 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.557300 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.557482 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.557578 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.557583 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.558824 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.558827 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.559385 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.559393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.559864 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.559867 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.559948 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.560505 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.560519 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.561963 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.563950 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.563954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.563956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.564168 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.564449 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.564454 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.564456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.564539 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.564930 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.564933 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.565041 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.565150 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.565416 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":118}
I20260928 14:46:17.565572 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.556828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51424 (local address 127.13.223.125:43405)
0928 14:46:17.557018 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.557024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.557036 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:46:17.557159 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.557162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.557199 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.557290 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.557690 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.558656 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.559513 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.559754 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.559864 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.562149 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.562152 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.562155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.562186 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.563799 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.564271 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.564276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.564278 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.564356 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.564626 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.564631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.564633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.564829 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.564932 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.565194 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.565362 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260928 14:46:17.569016 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Bootstrap starting.
I20260928 14:46:17.572263 20858 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:17.573464 20858 log.cc:824] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:17.576368 20858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: No bootstrap required, opened a new log
I20260928 14:46:17.579954 20858 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } }
I20260928 14:46:17.580262 20858 raft_consensus.cc:374] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:17.580302 20858 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d12c09dbbaf54e06b2cec5d0403c7d5f, State: Initialized, Role: FOLLOWER
I20260928 14:46:17.580914 20858 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } }
I20260928 14:46:17.581602 20864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } } }
I20260928 14:46:17.581781 20864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:17.582489 20858 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:46:17.589437 20875 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:46:17.589596 20875 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:46:17.592295 20790 master_runner.cc:428] Detected that this master d12c09dbbaf54e06b2cec5d0403c7d5f is joining an existing cluster
I20260928 14:46:17.592394 20790 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.223.125:43405
I20260928 14:46:17.605667 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.596467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51438 (local address 127.13.223.125:43405)
0928 14:46:17.596604 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.596610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.596943 (+ 333us) server_negotiation.cc:407] Connection header received
0928 14:46:17.597087 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.597091 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.597133 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.597218 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.597743 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.598515 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.599616 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.599833 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.599955 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.601564 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.601566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.601568 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.601587 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.603230 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604021 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.604024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.604026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.604127 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604696 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.604699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.604701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.604951 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.605065 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.605356 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.605530 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":80,"server-negotiator.queue_time_us":54}
I20260928 14:46:17.606369 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.595686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42528)
0928 14:46:17.596070 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.596087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.596216 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.596872 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.596875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.597151 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.597346 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.597354 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.598307 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.598313 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.599276 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.599286 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.599947 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.599950 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.600069 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.600694 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.600707 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.601287 (+ 580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.603814 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.603817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.603819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.604138 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.604481 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604486 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.604488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.604594 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.605861 (+ 1267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.605865 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.605954 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.606053 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.606197 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":26,"thread_start_us":150,"threads_started":1}
I20260928 14:46:17.606801 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.596478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51438)
0928 14:46:17.596839 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.596851 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.596986 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.597231 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.597234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.597497 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.597608 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.597615 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.598655 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.598658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.599509 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.599516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.600083 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.600086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.600184 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.600878 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.600894 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.601389 (+ 495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.603388 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.603394 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.603396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.603832 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.604495 (+ 663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.604500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.604601 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.605073 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.605077 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.605181 (+ 104us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.605287 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.606620 (+ 1333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":103,"threads_started":1}
I20260928 14:46:17.608741 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.595771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.13.223.126:44095)
0928 14:46:17.596028 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.596034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.596188 (+ 154us) server_negotiation.cc:407] Connection header received
0928 14:46:17.596568 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.596572 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.596638 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.596699 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.597502 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.598186 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.599423 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.599793 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.599916 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.601469 (+ 1553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.601473 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.601476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.601525 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.603691 (+ 2166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604248 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.604252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.604254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.604391 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.604723 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.604727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.604729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.604970 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.605091 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.608473 (+ 3382us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.608625 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
WARNING: no policy specified for oryx/127.13.223.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.124@KRBTEST.COM" created.
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.126@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
I20260928 14:46:17.627446 20861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.615354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.13.223.125:43405)
0928 14:46:17.615507 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.615513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.615534 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:46:17.615620 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.615625 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.615693 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.615802 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.616731 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.617300 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.618996 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.619219 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.619388 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.622740 (+ 3352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.622747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.622753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.622793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.624831 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.625573 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.625581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.625585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.625694 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.626048 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.626054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.626058 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.626437 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.626585 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.627082 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.627237 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:46:17.627646 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.614430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51450)
0928 14:46:17.614948 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.615017 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.615213 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.615877 (+ 664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.615884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.616400 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.616583 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.616593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.617606 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.617610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.618841 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.618856 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.619409 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.619413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.619566 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.620365 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.620384 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.622492 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.624986 (+ 2494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.625001 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.625010 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.625418 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.625812 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.625815 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.625818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.625926 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.626628 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.626637 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.626799 (+ 162us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.626974 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.627232 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
I20260928 14:46:17.629580 20325 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.223.125:43405: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
member_type: NON_VOTER
last_known_addr {
host: "127.13.223.125"
port: 43405
}
attrs {
promote: true
}
}
dest_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828"
cas_config_opid_index: -1
I20260928 14:46:17.630386 20325 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } }
I20260928 14:46:17.632555 20888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } }
I20260928 14:46:17.633543 20832 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:17.635159 20832 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 14:46:17.635571 20888 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.223.125:43405
I20260928 14:46:17.636231 20888 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:17.636689 20790 master.cc:537] Master@127.13.223.125:43405 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.124.keytab.
Entry for principal oryx/127.13.223.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.124.keytab.
W20260928 14:46:17.638608 20311 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 14:46:17.636260 20887 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } }
I20260928 14:46:17.638980 20887 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:17.639277 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:44517
--webserver_interface=127.13.223.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.124.keytab
--principal=oryx/127.13.223.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:17.641044 20311 consensus_peers.cc:487] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 -> Peer d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125:43405): Couldn't send request to peer d12c09dbbaf54e06b2cec5d0403c7d5f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:46:17.643131 20790 raft_consensus.cc:2229] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:17.643461 20790 raft_consensus.cc:2258] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:17.643589 20790 tablet_replica.cc:330] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: stopping tablet replica
I20260928 14:46:17.648758 20790 master.cc:552] Master@127.13.223.125:43405 shutdown complete.
I20260928 14:46:17.648850 20790 master_runner.cc:305] Clearing existing system tablet
I20260928 14:46:17.649094 20790 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:17.649166 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:17.650336 20790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260928 14:46:17.651293 20894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.651520 20790 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:17.651675 20790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:46:17.651712 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "49948943d49f59631f5f1b8606be6813"
server_key_iv: "3ded0ed3a74ef44c4bd617ebec82a8b8"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.653496 20790 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:17.655650 20790 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:46:17.655768 20790 log.cc:1196] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 14:46:17.656019 20790 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Deleting consensus metadata
I20260928 14:46:17.656220 20790 master_runner.cc:315] Copying system tablet from 127.13.223.126:44095
I20260928 14:46:17.657675 20790 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: tablet copy: Beginning tablet copy session from remote peer at address 127.13.223.126:44095
I20260928 14:46:17.666421 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.658373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42536 (local address 127.13.223.126:44095)
0928 14:46:17.658525 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.658532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.658784 (+ 252us) server_negotiation.cc:407] Connection header received
0928 14:46:17.658939 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.658942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.659001 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.659092 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.659558 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.660103 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.660956 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.661202 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.661321 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.662543 (+ 1222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.662569 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.662576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.662608 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.664327 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.664949 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.664953 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.664955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.665064 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.665409 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.665413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.665416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.665645 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.665771 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.666080 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.666211 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:46:17.666432 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.658315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42536)
0928 14:46:17.658686 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.658698 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.658836 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.659110 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.659114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.659327 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.659434 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.659440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.660221 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.660226 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.660813 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.660822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.661329 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.661333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.661435 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.662031 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.662043 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.662392 (+ 349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.664487 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.664493 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.664496 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.664802 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.665176 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.665180 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.665182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.665293 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.665756 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.665759 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.665891 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.666015 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.666214 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":102,"threads_started":1}
I20260928 14:46:17.667630 20345 tablet_copy_service.cc:140] P cda5c6fa51af4e8c81f35f3b6f1e8828: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d12c09dbbaf54e06b2cec5d0403c7d5f ({username='oryx', principal='oryx/127.13.223.125@KRBTEST.COM'} at 127.0.0.1:42536)
I20260928 14:46:17.667846 20345 tablet_copy_service.cc:161] P cda5c6fa51af4e8c81f35f3b6f1e8828: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d12c09dbbaf54e06b2cec5d0403c7d5f at {username='oryx', principal='oryx/127.13.223.125@KRBTEST.COM'} at 127.0.0.1:42536: session id = d12c09dbbaf54e06b2cec5d0403c7d5f-00000000000000000000000000000000
I20260928 14:46:17.669545 20345 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:46:17.671379 20790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:46:17.673825 20790 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: tablet copy: Starting download of 0 data blocks...
I20260928 14:46:17.674090 20790 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: tablet copy: Starting download of 1 WAL segments...
I20260928 14:46:17.676100 20790 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:46:17.678144 20345 tablet_copy_service.cc:342] P cda5c6fa51af4e8c81f35f3b6f1e8828: Request end of tablet copy session d12c09dbbaf54e06b2cec5d0403c7d5f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.223.125@KRBTEST.COM'} at 127.0.0.1:42536
I20260928 14:46:17.678324 20345 tablet_copy_service.cc:433] P cda5c6fa51af4e8c81f35f3b6f1e8828: ending tablet copy session d12c09dbbaf54e06b2cec5d0403c7d5f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d12c09dbbaf54e06b2cec5d0403c7d5f
I20260928 14:46:17.686995 20790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:17.689011 20904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.689333 20790 server_base.cc:1034] running on GCE node
W20260928 14:46:17.689339 20901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.690840 20902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.691120 20790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:17.691512 20790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:17.692664 20790 hybrid_clock.cc:648] HybridClock initialized: now 1790606777692678 us; error 55 us; skew 500 ppm
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:17.694401 20790 init.cc:377] Logged in from keytab as oryx/127.13.223.125@KRBTEST.COM (short username oryx)
I20260928 14:46:17.695050 20790 webserver.cc:460] Webserver started at http://127.13.223.125:35849/ using document root <none> and password file <none>
I20260928 14:46:17.695276 20790 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:17.695346 20790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:17.696753 20790 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:46:17.697818 20911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.698094 20790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260928 14:46:17.698171 20790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "49948943d49f59631f5f1b8606be6813"
server_key_iv: "3ded0ed3a74ef44c4bd617ebec82a8b8"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.698259 20790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:17.707808 20790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:17.709975 20790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:17.715167 20790 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.125:43405
I20260928 14:46:17.715529 20963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.125:43405 every 8 connection(s)
I20260928 14:46:17.715942 20790 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
I20260928 14:46:17.717680 20964 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:46:17.718719 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Bootstrap starting.
I20260928 14:46:17.727991 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:46:17.728323 20964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: Bootstrap complete.
I20260928 14:46:17.728870 20964 raft_consensus.cc:348] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } }
I20260928 14:46:17.729017 20964 raft_consensus.cc:729] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d12c09dbbaf54e06b2cec5d0403c7d5f, State: Initialized, Role: LEARNER
I20260928 14:46:17.729137 20964 consensus_queue.cc:260] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } }
I20260928 14:46:17.729521 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } }
I20260928 14:46:17.729687 20964 sys_catalog.cc:564] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:17.730003 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: LEARNER
I20260928 14:46:17.732933 20978 catalog_manager.cc:1259] Loaded cluster ID: dd07be5537b5430daf44285a88ecf056
I20260928 14:46:17.736605 20978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: loading cluster ID for follower catalog manager: success
I20260928 14:46:17.739153 20978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: acquiring CA information for follower catalog manager: success
I20260928 14:46:17.740873 20978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:46:17.760982 20516 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
W20260928 14:46:17.796072 20892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:17.796422 20892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:17.796526 20892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:17.801054 20892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:17.801153 20892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:17.801189 20892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:17.801230 20892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:17.801266 20892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:17.807173 20892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.124.keytab
--principal=oryx/127.13.223.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:44517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:17.808390 20892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:17.809487 20892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:17.818899 20986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:17.818953 20984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.819164 20892 server_base.cc:1034] running on GCE node
W20260928 14:46:17.818899 20983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:17.819924 20892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:17.821228 20892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:17.822443 20892 hybrid_clock.cc:648] HybridClock initialized: now 1790606777822425 us; error 33 us; skew 500 ppm
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:17.825635 20892 init.cc:377] Logged in from keytab as oryx/127.13.223.124@KRBTEST.COM (short username oryx)
I20260928 14:46:17.826542 20892 webserver.cc:460] Webserver started at http://127.13.223.124:34693/ using document root <none> and password file <none>
I20260928 14:46:17.827183 20892 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:17.827278 20892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:17.827527 20892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:17.829905 20892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/instance:
uuid: "a801913b65ef454fa708392b7df506ea"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "3c43fbda4da4d7459388447f4851d265"
server_key_iv: "ba5eedf01706e39f666e792544010f4b"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.830638 20892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal/instance:
uuid: "a801913b65ef454fa708392b7df506ea"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "3c43fbda4da4d7459388447f4851d265"
server_key_iv: "ba5eedf01706e39f666e792544010f4b"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.835212 20892 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:17.837939 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:17.839421 20892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:17.839581 20892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "a801913b65ef454fa708392b7df506ea"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "3c43fbda4da4d7459388447f4851d265"
server_key_iv: "ba5eedf01706e39f666e792544010f4b"
server_key_version: "encryptionkey@0"
I20260928 14:46:17.839706 20892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:17.851651 20892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:17.855173 20892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:17.855396 20892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:17.863508 20892 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.124:44517
I20260928 14:46:17.863793 21045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.124:44517 every 8 connection(s)
I20260928 14:46:17.864732 20892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
I20260928 14:46:17.867818 21046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:17.869782 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 20892
I20260928 14:46:17.872624 21046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.124@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:17.885658 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.874830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42546 (local address 127.13.223.126:44095)
0928 14:46:17.874984 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.874988 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.875227 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:46:17.875395 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.875398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.875451 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.875541 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.876356 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.877061 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.878241 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.878466 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.878640 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.882170 (+ 3530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.882174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.882176 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.882201 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.883615 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.884131 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.884134 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.884136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.884220 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.884538 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.884543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.884545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.884769 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.884864 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.885331 (+ 467us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.885464 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:46:17.886168 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.874666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42546)
0928 14:46:17.875104 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.875145 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.875283 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.875593 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.875599 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.876063 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.876224 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.876233 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.877214 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.877217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.878126 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.878133 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.879784 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.879790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.879945 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.880519 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.880538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.882027 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.883714 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.883725 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.883731 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.884024 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.884319 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.884322 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.884323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.884435 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.884890 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.884898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.885039 (+ 141us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.885262 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.885472 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":163,"threads_started":1}
I20260928 14:46:17.886171 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.871450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47954 (local address 127.13.223.124:44517)
0928 14:46:17.871918 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.871936 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.871972 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:46:17.872630 (+ 658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.872646 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.873064 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.873366 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.874301 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.875737 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.876833 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.877724 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.877880 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.881043 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.881048 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.881059 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.881081 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.882938 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.883403 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.883410 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.883416 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.883532 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.883878 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.883881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.883882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.884448 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.884682 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.884933 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.885240 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"spinlock_wait_cycles":1280,"thread_start_us":157,"threads_started":1}
I20260928 14:46:17.887504 21046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.124@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
I20260928 14:46:17.896721 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.888038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51460)
0928 14:46:17.888194 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.888211 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.888319 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.888804 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.888806 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.889014 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.889127 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.889132 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.890341 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.890344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.891006 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.891016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.891466 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.891469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.891542 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.892046 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.892057 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.893340 (+ 1283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.895099 (+ 1759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.895119 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.895121 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.895420 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.895686 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.895688 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.895690 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.895764 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.896209 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.896211 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.896297 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.896469 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.896592 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 14:46:17.896763 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.888222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.13.223.125:43405)
0928 14:46:17.888543 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.888560 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.888577 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:17.888624 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.888627 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.888679 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.888779 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.889257 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.890236 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.891115 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.891368 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.891459 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.893474 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.893478 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.893480 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.893504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.894985 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.895513 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.895516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.895518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.895601 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.895878 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.895881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.895882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.896097 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.896193 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.896526 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.896638 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":107,"threads_started":1}
I20260928 14:46:17.897631 21046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606777, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.124@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
I20260928 14:46:17.906201 21049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.898203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:47970)
0928 14:46:17.898342 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.898359 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.898491 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.898788 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.898791 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.898965 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.899035 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.899040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.900181 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.900186 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.900746 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.900755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.901210 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.901213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.901277 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.901775 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.901786 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.903260 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.904856 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.904859 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.904861 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.905057 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.905311 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.905314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.905315 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.905394 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.905746 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.905749 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.905825 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.905909 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.906036 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:46:17.906224 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.898232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47970 (local address 127.13.223.124:44517)
0928 14:46:17.898342 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.898348 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.898424 (+ 76us) server_negotiation.cc:407] Connection header received
0928 14:46:17.898650 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.898655 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.898693 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.898779 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.899142 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.900077 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.900869 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.901109 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.901195 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.903423 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.903434 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.903437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.903458 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.904741 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.905141 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.905143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.905144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.905213 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.905475 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.905477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.905479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.905669 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.905745 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.905958 (+ 213us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.906098 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":46}
I20260928 14:46:17.909390 21046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Bootstrap starting.
I20260928 14:46:17.911988 21046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:17.912762 21046 log.cc:824] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:17.914983 21046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: No bootstrap required, opened a new log
I20260928 14:46:17.918066 21046 raft_consensus.cc:348] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } }
I20260928 14:46:17.918401 21046 raft_consensus.cc:374] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:17.918460 21046 raft_consensus.cc:729] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a801913b65ef454fa708392b7df506ea, State: Initialized, Role: FOLLOWER
I20260928 14:46:17.918982 21046 consensus_queue.cc:260] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } }
I20260928 14:46:17.919533 21052 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } } }
I20260928 14:46:17.919780 21052 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:17.920328 21046 sys_catalog.cc:564] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:46:17.925117 21063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:46:17.925242 21063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:46:17.929351 20892 master_runner.cc:428] Detected that this master a801913b65ef454fa708392b7df506ea is joining an existing cluster
I20260928 14:46:17.929450 20892 master_runner.cc:432] Initiating AddMaster RPC to add 127.13.223.124:44517
I20260928 14:46:17.941238 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20387
I20260928 14:46:17.949143 21071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.932368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42550)
0928 14:46:17.933304 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.933319 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.933418 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.933676 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.933678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.933880 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.934076 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.934084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.935398 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.935403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.936196 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.936211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.936929 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.936932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.937012 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.937661 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.937674 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.938102 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.946188 (+ 8086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.946194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.946197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.946807 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.947452 (+ 645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.947459 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.947464 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.947631 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.948398 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.948404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.948531 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.948665 (+ 134us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.948859 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":115,"threads_started":1}
I20260928 14:46:17.949779 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.932624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42550 (local address 127.13.223.126:44095)
0928 14:46:17.932867 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.932873 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.933379 (+ 506us) server_negotiation.cc:407] Connection header received
0928 14:46:17.933518 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.933521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.933577 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.933670 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.934257 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.935205 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.936343 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.936702 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.936817 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.938293 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.938297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.938301 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.938330 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.946048 (+ 7718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.947189 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.947193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.947196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.947316 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.947757 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.947763 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.947765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.948136 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.948261 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.949450 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.949598 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170}
I20260928 14:46:17.950476 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:41187
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:44095
--webserver_port=44287
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:46:17.953295 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.933038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51464 (local address 127.13.223.125:43405)
0928 14:46:17.933162 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.933167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.939303 (+ 6136us) server_negotiation.cc:407] Connection header received
0928 14:46:17.939834 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.939838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.939902 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.939966 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.940868 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.941813 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.943958 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.944387 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.944483 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.947529 (+ 3046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.947534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.947537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.947572 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.950245 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.951234 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.951237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.951239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.951333 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.951669 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.951672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.951674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.951943 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.952029 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.952896 (+ 867us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.953076 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:46:17.954005 21070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.932942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51464)
0928 14:46:17.939211 (+ 6269us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.939225 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.939379 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.940366 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.940369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.940591 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.940707 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.940714 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.942366 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.942369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.943790 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.943803 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.944942 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.944947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.945086 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.946265 (+ 1179us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.946281 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.947350 (+ 1069us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.950858 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.950861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.950863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.951110 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.951454 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.951457 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.951459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.951563 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.952091 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.952096 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.952519 (+ 423us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.953687 (+ 1168us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.953832 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6192,"thread_start_us":113,"threads_started":1}
Sep 28 14:46:17 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606776, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.126@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
I20260928 14:46:17.966377 21048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.932454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.13.223.124:44517)
0928 14:46:17.932767 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.932775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.939042 (+ 6267us) server_negotiation.cc:407] Connection header received
0928 14:46:17.939201 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.939204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.939258 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.939330 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.940078 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.940910 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.942216 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.942608 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.942687 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.944386 (+ 1699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.944391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.944394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.944422 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.964659 (+ 20237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965175 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.965179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.965181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.965248 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965493 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.965496 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.965498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.965713 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.965777 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.966070 (+ 293us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.966204 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260928 14:46:17.966732 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.954021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:47990)
0928 14:46:17.954185 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.954199 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.954270 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.954870 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.954874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.955073 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.955189 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.955195 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.956320 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.956324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.957237 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.957248 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.957925 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.957929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.958036 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.958691 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.958713 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.960472 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.964954 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.964959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.964961 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.965242 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.965577 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965580 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.965582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.965672 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.966114 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.966118 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.966223 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.966304 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.966500 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 14:46:17.967432 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.954116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.13.223.124:44517)
0928 14:46:17.954508 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.954515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.954526 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:46:17.954665 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.954668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.954723 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.954819 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:17.955323 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.956173 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.957390 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.957788 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.958295 (+ 507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.960655 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:17.960659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:17.960662 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:17.960693 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:17.964792 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965371 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.965374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.965376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.965469 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965781 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:17.965784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:17.965786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:17.965999 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:17.967020 (+ 1021us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.967120 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.967250 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":105,"threads_started":1}
I20260928 14:46:17.968080 21069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.932394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:47980)
0928 14:46:17.938935 (+ 6541us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.938954 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.939083 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.939505 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.939509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.939808 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:17.939958 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.939965 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.941029 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.941033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.942082 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.942097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.942874 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.942878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.942999 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.943615 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:17.943631 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:17.944216 (+ 585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:17.964821 (+ 20605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.964824 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.964826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.965076 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.965339 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:17.965342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:17.965344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:17.965413 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:17.965843 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:17.965845 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:17.965921 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.967393 (+ 1472us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.967741 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6427,"thread_start_us":87,"threads_started":1}
I20260928 14:46:17.968515 20325 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.13.223.124:44517: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a801913b65ef454fa708392b7df506ea"
member_type: NON_VOTER
last_known_addr {
host: "127.13.223.124"
port: 44517
}
attrs {
promote: true
}
}
dest_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828"
cas_config_opid_index: 5
I20260928 14:46:17.969179 20325 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } }
I20260928 14:46:17.970340 20888 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a801913b65ef454fa708392b7df506ea (127.13.223.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.971216 21020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:17.971938 20888 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.13.223.124:44517
I20260928 14:46:17.972138 20888 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.972265 20888 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:17.972744 20892 master.cc:537] Master@127.13.223.124:44517 shutting down...
I20260928 14:46:17.972836 21020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:46:17.973716 20887 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:46:17.975746 20312 consensus_peers.cc:487] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 -> Peer a801913b65ef454fa708392b7df506ea (127.13.223.124:44517): Couldn't send request to peer a801913b65ef454fa708392b7df506ea. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 14:46:17.978793 20892 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:17.979221 20892 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:17.979393 20892 tablet_replica.cc:330] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: stopping tablet replica
I20260928 14:46:17.979696 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.970333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51472)
0928 14:46:17.970608 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.970623 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:17.970702 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:17.974396 (+ 3694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:17.974400 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:17.974431 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:17.974541 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.974582 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.975646 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.975649 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.976987 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:17.977000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.979175 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.979180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:17.979293 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.979356 (+ 63us) client_negotiation.cc:769] Sending connection context
0928 14:46:17.979447 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:46:17.979541 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260928 14:46:17.980189 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.970741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51472 (local address 127.13.223.125:43405)
0928 14:46:17.973803 (+ 3062us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:17.973809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:17.973826 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:17.973883 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:17.973886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:17.973959 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:17.974068 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:17.974719 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.975515 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.978101 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:17.978835 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:17.978960 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:17.979103 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:17.979978 (+ 875us) server_negotiation.cc:299] Negotiation successful
0928 14:46:17.980070 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2951}
I20260928 14:46:17.980657 20938 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 LEARNER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 14:46:17.981094 20887 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 14:46:17.983253 20938 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER a801913b65ef454fa708392b7df506ea (127.13.223.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.985680 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5c6fa51af4e8c81f35f3b6f1e8828. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.985821 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: LEARNER
I20260928 14:46:17.985912 20967 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: NON_VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: true } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.985986 20967 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: LEARNER
I20260928 14:46:17.987502 20887 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: attempting to promote NON_VOTER d12c09dbbaf54e06b2cec5d0403c7d5f to VOTER
I20260928 14:46:17.987994 20887 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } }
I20260928 14:46:17.989728 20937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 LEARNER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 14:46:17.990154 20889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 14:46:17.990931 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.989071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:47998)
0928 14:46:17.990746 (+ 1675us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.990845 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1583}
I20260928 14:46:17.991875 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:17.991621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:48008)
0928 14:46:17.991741 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:17.991813 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:44517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:46:17.991972 20312 consensus_peers.cc:487] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 -> Peer a801913b65ef454fa708392b7df506ea (127.13.223.124:44517): Couldn't send request to peer a801913b65ef454fa708392b7df506ea. Status: Network error: Client connection negotiation failed: client connection to 127.13.223.124:44517: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:46:17.992637 20889 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.994217 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.994335 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:17.994472 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.995177 20937 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.996912 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:17.997102 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:17.997293 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:17.999704 20892 master.cc:552] Master@127.13.223.124:44517 shutdown complete.
I20260928 14:46:17.999758 20892 master_runner.cc:305] Clearing existing system tablet
I20260928 14:46:18.000027 20892 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:18.000075 20892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:18.006800 20892 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.000s
I20260928 14:46:18.007854 21077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.008123 20892 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:18.008312 20892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:46:18.008366 20892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "a801913b65ef454fa708392b7df506ea"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "3c43fbda4da4d7459388447f4851d265"
server_key_iv: "ba5eedf01706e39f666e792544010f4b"
server_key_version: "encryptionkey@0"
I20260928 14:46:18.010676 20892 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:18.013273 20892 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 14:46:18.013435 20892 log.cc:1196] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 14:46:18.013721 20892 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Deleting consensus metadata
I20260928 14:46:18.013929 20892 master_runner.cc:315] Copying system tablet from 127.13.223.126:44095
I20260928 14:46:18.015429 20892 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: tablet copy: Beginning tablet copy session from remote peer at address 127.13.223.126:44095
I20260928 14:46:18.025545 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.016151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42554 (local address 127.13.223.126:44095)
0928 14:46:18.016292 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.016298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.016462 (+ 164us) server_negotiation.cc:407] Connection header received
0928 14:46:18.016577 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.016580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.016633 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.016730 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.017276 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.018165 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.019522 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.019886 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.020000 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.021701 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.021706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.021709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.021739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.023608 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.024151 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.024155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.024158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.024248 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.024544 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.024548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.024549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.024781 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.024856 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.025233 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.025372 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:46:18.026278 21083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.016048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42554)
0928 14:46:18.016357 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.016372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.016490 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.016763 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.016766 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.017001 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.017147 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.017154 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.018296 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.018302 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.019384 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.019398 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.020039 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.020043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.020150 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.020941 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.020957 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.021467 (+ 510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.023749 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.023753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.023755 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.024031 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.024357 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.024360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.024362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.024443 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.024928 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.024931 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.025045 (+ 114us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.025897 (+ 852us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.026075 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":88,"threads_started":1}
I20260928 14:46:18.026858 20345 tablet_copy_service.cc:140] P cda5c6fa51af4e8c81f35f3b6f1e8828: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a801913b65ef454fa708392b7df506ea ({username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:42554)
I20260928 14:46:18.026968 20345 tablet_copy_service.cc:161] P cda5c6fa51af4e8c81f35f3b6f1e8828: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a801913b65ef454fa708392b7df506ea at {username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:42554: session id = a801913b65ef454fa708392b7df506ea-00000000000000000000000000000000
I20260928 14:46:18.027433 20651 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
I20260928 14:46:18.028096 20345 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 14:46:18.029484 20892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 14:46:18.032092 20892 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: tablet copy: Starting download of 0 data blocks...
I20260928 14:46:18.032380 20892 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: tablet copy: Starting download of 1 WAL segments...
I20260928 14:46:18.034178 20892 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 14:46:18.036141 20345 tablet_copy_service.cc:342] P cda5c6fa51af4e8c81f35f3b6f1e8828: Request end of tablet copy session a801913b65ef454fa708392b7df506ea-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:42554
I20260928 14:46:18.036232 20345 tablet_copy_service.cc:433] P cda5c6fa51af4e8c81f35f3b6f1e8828: ending tablet copy session a801913b65ef454fa708392b7df506ea-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a801913b65ef454fa708392b7df506ea
I20260928 14:46:18.044737 20892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:18.046977 21084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.047075 21087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.047435 21085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:18.047745 20892 server_base.cc:1034] running on GCE node
I20260928 14:46:18.047964 20892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:18.048285 20892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:18.049439 20892 hybrid_clock.cc:648] HybridClock initialized: now 1790606778049426 us; error 32 us; skew 500 ppm
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:18.051203 20892 init.cc:377] Logged in from keytab as oryx/127.13.223.124@KRBTEST.COM (short username oryx)
I20260928 14:46:18.051720 20892 webserver.cc:460] Webserver started at http://127.13.223.124:38527/ using document root <none> and password file <none>
I20260928 14:46:18.051934 20892 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:18.051977 20892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:18.053520 20892 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:46:18.054764 21094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.055133 20892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 14:46:18.055214 20892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "a801913b65ef454fa708392b7df506ea"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "3c43fbda4da4d7459388447f4851d265"
server_key_iv: "ba5eedf01706e39f666e792544010f4b"
server_key_version: "encryptionkey@0"
I20260928 14:46:18.055307 20892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:18.080428 20892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:18.083027 20892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:18.088824 20892 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.124:44517
I20260928 14:46:18.089488 20892 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
I20260928 14:46:18.090082 21146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.124:44517 every 8 connection(s)
I20260928 14:46:18.091747 21147 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:46:18.093155 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Bootstrap starting.
W20260928 14:46:18.105903 21072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:18.106244 21072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:18.106325 21072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:46:18.107515 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:46:18.107899 21147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Bootstrap complete.
I20260928 14:46:18.108593 21147 raft_consensus.cc:348] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } }
I20260928 14:46:18.108757 21147 raft_consensus.cc:729] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a801913b65ef454fa708392b7df506ea, State: Initialized, Role: LEARNER
I20260928 14:46:18.108883 21147 consensus_queue.cc:260] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } }
I20260928 14:46:18.109278 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:18.109397 21147 sys_catalog.cc:564] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:18.109503 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: LEARNER
W20260928 14:46:18.111802 21072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:18.111912 21072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:18.112025 21072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:18.114338 21164 catalog_manager.cc:1259] Loaded cluster ID: dd07be5537b5430daf44285a88ecf056
I20260928 14:46:18.114413 21164 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: loading cluster ID for follower catalog manager: success
I20260928 14:46:18.116744 21164 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: acquiring CA information for follower catalog manager: success
I20260928 14:46:18.118350 21164 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:46:18.120337 21072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:41187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=44287
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:18.121891 21072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:18.123172 21072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:18.135389 21169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.135389 21166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.135466 21167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:18.135802 21072 server_base.cc:1034] running on GCE node
I20260928 14:46:18.136288 21072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:18.137079 21072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:18.138294 21072 hybrid_clock.cc:648] HybridClock initialized: now 1790606778138248 us; error 36 us; skew 500 ppm
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:18.141554 21072 init.cc:377] Logged in from keytab as oryx/127.13.223.65@KRBTEST.COM (short username oryx)
I20260928 14:46:18.142544 21072 webserver.cc:460] Webserver started at http://127.13.223.65:44287/ using document root <none> and password file <none>
I20260928 14:46:18.143198 21072 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:18.143275 21072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:18.147878 21072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:18.150465 21176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.152087 21072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:18.152252 21072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "0c83b31373ec423c8f4524befc90bfc4"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "d4d352b1a68afe5fba7102ea0f073b0c"
server_key_iv: "3cf42e9bd76ce155a17416f27608f286"
server_key_version: "encryptionkey@0"
I20260928 14:46:18.152890 21072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:18.178707 21072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:18.182427 21072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:18.182677 21072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:18.183326 21072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:18.184335 21072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:18.184422 21072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.184499 21072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:18.184541 21072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.194608 21072 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:41187
I20260928 14:46:18.194626 21289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:41187 every 8 connection(s)
I20260928 14:46:18.195819 21072 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:18.201431 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21072
I20260928 14:46:18.201674 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20521
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:18.213740 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:41435
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:44095
--webserver_port=33475
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:46:18.216436 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.197944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:48335 (local address 127.13.223.124:44517)
0928 14:46:18.198291 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.198295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.199186 (+ 891us) server_negotiation.cc:407] Connection header received
0928 14:46:18.200574 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.200578 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.200659 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.200751 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.203909 (+ 3158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.205060 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206252 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.206683 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206770 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.210723 (+ 3953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.210729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.210732 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.210772 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.212991 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.213792 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.213797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.213799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.213902 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214764 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.214768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.214770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.215047 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.215133 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.216096 (+ 963us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.216241 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":99,"threads_started":1}
I20260928 14:46:18.216434 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.199741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:50767 (local address 127.13.223.125:43405)
0928 14:46:18.199860 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.199863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.200083 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:46:18.200880 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.200884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.200939 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.201042 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.202373 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.203308 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.205389 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.205741 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.205824 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.211671 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.211678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.211682 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.211724 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.213616 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214249 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.214253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.214256 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.214360 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214764 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.214767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.214769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.215047 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.215133 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.216067 (+ 934us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.216239 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:46:18.216485 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.197996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:33409 (local address 127.13.223.126:44095)
0928 14:46:18.198137 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.198142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.199188 (+ 1046us) server_negotiation.cc:407] Connection header received
0928 14:46:18.200458 (+ 1270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.200462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.200536 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.200610 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.202478 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.203494 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.205463 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.205975 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206088 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.211180 (+ 5092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.211188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.211195 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.211236 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.213697 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214395 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.214403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.214408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.214515 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214944 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.214951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.214956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.215241 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.215337 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.216093 (+ 756us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.216259 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:46:18.218647 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.199609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.13.223.65:50767)
0928 14:46:18.199980 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.199993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.200228 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.201095 (+ 867us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.201099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.201381 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.202181 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.202219 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.203503 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.203507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.205261 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.205269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.206072 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.206330 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.207156 (+ 826us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.207183 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.211430 (+ 4247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.213766 (+ 2336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.213770 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.213772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.214117 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.214501 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214504 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.214506 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.214642 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.215237 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.215248 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.215719 (+ 471us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.216772 (+ 1053us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.217237 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":183,"spinlock_wait_cycles":38656,"thread_start_us":96,"threads_started":1}
I20260928 14:46:18.218775 21295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.198442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:33409)
0928 14:46:18.199016 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.199070 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.200227 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.200832 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.200840 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.201359 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.202326 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.202336 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.203652 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.203659 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.205296 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.205312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.206286 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.206442 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.208160 (+ 1718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.208183 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.210978 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.213859 (+ 2881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.213865 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.213870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.214250 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.214698 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.214710 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.214815 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.215433 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.215440 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.215699 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.216681 (+ 982us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.217362 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":669,"spinlock_wait_cycles":53376,"thread_start_us":137,"threads_started":1}
I20260928 14:46:18.219496 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.198463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.13.223.65:48335)
0928 14:46:18.199041 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.199079 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.200215 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.200830 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.200838 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.203602 (+ 2764us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.203748 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.203754 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.205216 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.205218 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.206100 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.206107 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.206993 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.206997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.207067 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.207472 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.207483 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.210327 (+ 2844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.213167 (+ 2840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.213178 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.213204 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.213653 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.214348 (+ 695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.214351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.214353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.214636 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.215419 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.215422 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.215716 (+ 294us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.217258 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.217426 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":92,"spinlock_wait_cycles":101120,"thread_start_us":178,"threads_started":1}
I20260928 14:46:18.220498 21291 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:18.220525 21290 heartbeater.cc:347] Connected to a master server at 127.13.223.124:44517
I20260928 14:46:18.220551 21292 heartbeater.cc:347] Connected to a master server at 127.13.223.125:43405
I20260928 14:46:18.220804 21290 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.220804 21292 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.220808 21291 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.221655 21290 heartbeater.cc:511] Master 127.13.223.124:44517 requested a full tablet report, sending...
I20260928 14:46:18.221659 21292 heartbeater.cc:511] Master 127.13.223.125:43405 requested a full tablet report, sending...
I20260928 14:46:18.221889 21291 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:18.223704 21103 ts_manager.cc:194] Registered new tserver with Master: 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187)
I20260928 14:46:18.223798 20325 ts_manager.cc:194] Re-registered known tserver with Master: 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187)
I20260928 14:46:18.223876 20928 ts_manager.cc:194] Registered new tserver with Master: 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187)
I20260928 14:46:18.224797 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:33409
I20260928 14:46:18.281297 20785 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
W20260928 14:46:18.355129 21298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:18.355461 21298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:18.355508 21298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:18.360028 21298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:18.360116 21298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:18.360183 21298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:18.366194 21298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:41435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=33475
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:18.367686 21298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:18.369019 21298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:18.381256 21303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.381361 21304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.381340 21306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:18.381795 21298 server_base.cc:1034] running on GCE node
I20260928 14:46:18.382381 21298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:18.383270 21298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:18.386704 21298 hybrid_clock.cc:648] HybridClock initialized: now 1790606778386610 us; error 123 us; skew 500 ppm
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:18.398895 21298 init.cc:377] Logged in from keytab as oryx/127.13.223.66@KRBTEST.COM (short username oryx)
I20260928 14:46:18.400182 21298 webserver.cc:460] Webserver started at http://127.13.223.66:33475/ using document root <none> and password file <none>
I20260928 14:46:18.400873 21298 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:18.400978 21298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:18.406065 21298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:18.409638 21313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.411755 21298 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260928 14:46:18.411991 21298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "504a5636894b4f94bbae983fd2717e57"
format_stamp: "Formatted at 2026-09-28 14:46:16 on dist-test-slave-gkhx"
server_key: "1175a98399f9ff93f17ee9f21906ff70"
server_key_iv: "a78c90af015b47f57708b94a07d26512"
server_key_version: "encryptionkey@0"
I20260928 14:46:18.412653 21298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:18.424737 21298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:18.428467 21298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:18.428700 21298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:18.429478 21298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:18.430702 21298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:18.430766 21298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.430842 21298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:18.430876 21298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.442152 21298 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:41435
I20260928 14:46:18.442174 21426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:41435 every 8 connection(s)
I20260928 14:46:18.443467 21298 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:18.443984 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21298
I20260928 14:46:18.444196 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20656
I20260928 14:46:18.454020 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.448743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:48012)
0928 14:46:18.448982 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.448999 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.449095 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.449452 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.449459 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.449489 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.449646 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.449657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.450894 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.450900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.452258 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.452275 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.453487 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.453495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.453617 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.453648 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.453736 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.453858 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141}
I20260928 14:46:18.454172 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:45707
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:44095
--webserver_port=41251
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:44311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:46:18.454070 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.448833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48012 (local address 127.13.223.124:44517)
0928 14:46:18.449092 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.449097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.449114 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:18.449202 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.449205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.449277 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.449416 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.449813 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.450749 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.452484 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.453271 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.453432 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.453629 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.453787 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.453895 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":84,"threads_started":1}
I20260928 14:46:18.455552 21113 raft_consensus.cc:1203] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 14:46:18.456194 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda5c6fa51af4e8c81f35f3b6f1e8828. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:18.456401 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: LEARNER
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
I20260928 14:46:18.468667 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.452343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44595 (local address 127.13.223.125:43405)
0928 14:46:18.452484 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.452488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.453390 (+ 902us) server_negotiation.cc:407] Connection header received
0928 14:46:18.453533 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.453537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.453596 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.453659 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.454500 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.455451 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.456770 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457165 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.457251 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.462821 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.462828 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.462833 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.462885 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.465344 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466119 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.466124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.466127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.466242 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466827 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.466832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.466834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.467130 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.467222 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.468202 (+ 980us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.468386 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:46:18.468713 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.446163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:39761 (local address 127.13.223.124:44517)
0928 14:46:18.446348 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.446356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.450265 (+ 3909us) server_negotiation.cc:407] Connection header received
0928 14:46:18.450460 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.450464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.450514 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.450602 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.452452 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.453413 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.456625 (+ 3212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457029 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.457138 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.462821 (+ 5683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.462829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.462835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.462879 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.465137 (+ 2258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466102 (+ 965us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.466107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.466111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.466230 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466838 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.466844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.466848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.467285 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.467414 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.468227 (+ 813us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.468436 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":68}
I20260928 14:46:18.471021 21432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.446682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.13.223.66:39761)
0928 14:46:18.450175 (+ 3493us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.450191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.450337 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.450740 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.450743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.451381 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.452239 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.452267 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.454527 (+ 2260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.454532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.456258 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.456271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457356 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.457362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.457616 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.458678 (+ 1062us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.458708 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.462450 (+ 3742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.465314 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.465324 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.465350 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.465942 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.466375 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.466386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.466705 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.467501 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.467508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.467949 (+ 441us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.468927 (+ 978us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.469535 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3396,"mutex_wait_us":83,"spinlock_wait_cycles":35328,"thread_start_us":163,"threads_started":1}
I20260928 14:46:18.471019 21434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.451917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.13.223.66:44595)
0928 14:46:18.453259 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.453278 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.453399 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.453883 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.453886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.454145 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.454298 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.454306 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.455612 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.455618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.456620 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.456632 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457469 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.457472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.457615 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.458690 (+ 1075us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.458720 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.462446 (+ 3726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.465523 (+ 3077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.465528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.465533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.465966 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.466397 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.466401 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.466404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.466691 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.467326 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.467335 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.467846 (+ 511us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.468912 (+ 1066us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.469657 (+ 745us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":300,"spinlock_wait_cycles":7424,"thread_start_us":95,"threads_started":1}
I20260928 14:46:18.472904 21427 heartbeater.cc:347] Connected to a master server at 127.13.223.124:44517
I20260928 14:46:18.473193 21427 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.473764 21427 heartbeater.cc:511] Master 127.13.223.124:44517 requested a full tablet report, sending...
I20260928 14:46:18.474303 21429 heartbeater.cc:347] Connected to a master server at 127.13.223.125:43405
I20260928 14:46:18.474411 21429 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.474666 21429 heartbeater.cc:511] Master 127.13.223.125:43405 requested a full tablet report, sending...
I20260928 14:46:18.475450 21103 ts_manager.cc:194] Registered new tserver with Master: 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
I20260928 14:46:18.475580 20928 ts_manager.cc:194] Registered new tserver with Master: 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:18.482321 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.445889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:33523 (local address 127.13.223.126:44095)
0928 14:46:18.446047 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.446053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.448532 (+ 2479us) server_negotiation.cc:407] Connection header received
0928 14:46:18.449909 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.449916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.449977 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.450060 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.456722 (+ 6662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457669 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.470738 (+ 13069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.471150 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.471302 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.478327 (+ 7025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.478332 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.478336 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.478375 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.480312 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.480900 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.480904 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.480907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.481012 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.481369 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.481372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.481374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.481636 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.481732 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.482016 (+ 284us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.482155 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:46:18.482331 21431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.446468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:33523)
0928 14:46:18.448305 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.448360 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.449644 (+ 1284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.450443 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.450458 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.451323 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.456575 (+ 5252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.456583 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.457896 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.457902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.464209 (+ 6307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.464251 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.471488 (+ 7237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.471493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.471616 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.472438 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.472455 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.478121 (+ 5666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.480470 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.480472 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.480474 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.480716 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.481157 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.481160 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.481161 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.481261 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.481734 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.481736 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.481839 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.481927 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.482147 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1623,"mutex_wait_us":52,"thread_start_us":152,"threads_started":1}
I20260928 14:46:18.483489 21428 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:18.483640 21428 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.483911 21428 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:18.484745 20325 ts_manager.cc:194] Re-registered known tserver with Master: 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
I20260928 14:46:18.485548 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:33523
W20260928 14:46:18.602192 21435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:18.602788 21435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:18.602885 21435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:18.610364 21435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:18.610473 21435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:18.610626 21435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:18.620505 21435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:44311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:45707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=41251
--tserver_master_addrs=127.13.223.126:44095,127.13.223.125:43405,127.13.223.124:44517
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:18.622288 21435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:18.623919 21435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:18.636029 21441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.635980 21443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:18.636515 21440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:18.637606 21435 server_base.cc:1034] running on GCE node
I20260928 14:46:18.638187 21435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:18.639073 21435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:18.640331 21435 hybrid_clock.cc:648] HybridClock initialized: now 1790606778640266 us; error 75 us; skew 500 ppm
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:18.643996 21435 init.cc:377] Logged in from keytab as oryx/127.13.223.67@KRBTEST.COM (short username oryx)
I20260928 14:46:18.645049 21435 webserver.cc:460] Webserver started at http://127.13.223.67:41251/ using document root <none> and password file <none>
I20260928 14:46:18.645699 21435 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:18.645805 21435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:18.650224 21435 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:18.652598 21450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.654045 21435 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:18.654208 21435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "4b8612fb089f409d9faffefe7c9cdec4"
format_stamp: "Formatted at 2026-09-28 14:46:17 on dist-test-slave-gkhx"
server_key: "dc9815be7099cc8f724fd402222ceb13"
server_key_iv: "45a74954d4e0073a447e50c33ded7abe"
server_key_version: "encryptionkey@0"
I20260928 14:46:18.655061 21435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:18.678164 21435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:18.682158 21435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:18.682406 21435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:18.683157 21435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:18.684245 21435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:18.684330 21435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.684412 21435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:18.684469 21435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:18.695175 21435 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:45707
I20260928 14:46:18.695243 21563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:45707 every 8 connection(s)
I20260928 14:46:18.696442 21435 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:18.700157 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21435
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.124@KRBTEST.COM
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.125@KRBTEST.COM
Sep 28 14:46:18 dist-test-slave-gkhx krb5kdc[20280](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606778, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:18.717620 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.698674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:50599 (local address 127.13.223.124:44517)
0928 14:46:18.698835 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.698841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.700275 (+ 1434us) server_negotiation.cc:407] Connection header received
0928 14:46:18.702124 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.702129 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.702199 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.702268 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.704524 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.705409 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707284 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.707644 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707713 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.712529 (+ 4816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.712533 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.712536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.712572 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.715185 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.715734 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.715738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.715740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.715842 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.716228 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.716232 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.716233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.716503 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.716582 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.717303 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.717442 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:46:18.717969 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.702915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48016 (local address 127.13.223.124:44517)
0928 14:46:18.703050 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.703056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.704294 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 14:46:18.704515 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.704518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.704571 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.704622 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.705645 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.706387 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707550 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.707840 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707907 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.714076 (+ 6169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.714081 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.714085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.714120 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.715974 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.716554 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.716558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.716560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.716634 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.716956 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.716959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.716960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.717159 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.717265 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.717706 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.717839 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:46:18.718840 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.700622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.13.223.67:60841)
0928 14:46:18.701258 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.701274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.701848 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.702729 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.702740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.703446 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.704361 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.704387 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.707100 (+ 2713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.708040 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.708051 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.709236 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.709340 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.709871 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.709888 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.711802 (+ 1914us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.714966 (+ 3164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.714976 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.715001 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.715536 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.715898 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.715901 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.715903 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.716111 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.716645 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.716651 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.717013 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.717233 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.717642 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":79,"spinlock_wait_cycles":26496,"thread_start_us":128,"threads_started":1}
I20260928 14:46:18.719383 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.699348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.13.223.67:50599)
0928 14:46:18.700068 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.700142 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.701822 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.702731 (+ 909us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.702742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.703440 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.704361 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.704385 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.705621 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.705627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.707141 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.707161 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.707952 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.707957 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.708189 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.708989 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.709012 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.712345 (+ 3333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.715320 (+ 2975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.715325 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.715330 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.715611 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.715961 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.715965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.715966 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.716114 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.716707 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.716710 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.717013 (+ 303us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.717871 (+ 858us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.718033 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":132,"spinlock_wait_cycles":8576,"thread_start_us":158,"threads_started":1}
I20260928 14:46:18.719826 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.700737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:60841 (local address 127.13.223.125:43405)
0928 14:46:18.700886 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.700892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.701350 (+ 458us) server_negotiation.cc:407] Connection header received
0928 14:46:18.702214 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.702218 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.702276 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.702354 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.704568 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.705414 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.708606 (+ 3192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.708997 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709108 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.712079 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.712084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.712087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.712121 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.714814 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.715674 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.715678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.715681 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.715784 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.716233 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.716237 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.716239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.716491 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.716583 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.719392 (+ 2809us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.719678 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:46:18.720745 21566 heartbeater.cc:347] Connected to a master server at 127.13.223.125:43405
I20260928 14:46:18.720975 21564 heartbeater.cc:347] Connected to a master server at 127.13.223.124:44517
I20260928 14:46:18.720996 21566 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.721086 21564 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.721621 21566 heartbeater.cc:511] Master 127.13.223.125:43405 requested a full tablet report, sending...
I20260928 14:46:18.721622 21564 heartbeater.cc:511] Master 127.13.223.124:44517 requested a full tablet report, sending...
I20260928 14:46:18.722854 20928 ts_manager.cc:194] Registered new tserver with Master: 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:18.722858 21103 ts_manager.cc:194] Registered new tserver with Master: 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:18.724550 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.699348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:35995)
0928 14:46:18.700938 (+ 1590us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.700957 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.701848 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.702729 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.702740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.703447 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:18.705634 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.705647 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.706849 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.706853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.708971 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.708982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.709794 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709802 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.709906 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.710419 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:18.710441 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:18.712284 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:18.722610 (+ 10326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.722614 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.722616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.722991 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.723386 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:18.723389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:18.723392 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:18.723498 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:18.724012 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:18.724015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:18.724112 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.724193 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.724350 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1485,"mutex_wait_us":2669,"spinlock_wait_cycles":62976,"thread_start_us":174,"threads_started":1}
I20260928 14:46:18.725057 21575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.702470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.13.223.126:44095)
0928 14:46:18.703043 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.703049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.704993 (+ 1944us) server_negotiation.cc:407] Connection header received
0928 14:46:18.705176 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.705179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.705234 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.705295 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.706403 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.707259 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709188 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.709942 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.710031 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.712695 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.712699 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.712702 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.712730 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.715720 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.723571 (+ 7851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.723575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.723577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.723680 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.724007 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.724010 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.724012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.724214 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.724333 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.724720 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.724888 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":101,"threads_started":1}
I20260928 14:46:18.725291 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.698726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:35995 (local address 127.13.223.126:44095)
0928 14:46:18.698842 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.698848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.701074 (+ 2226us) server_negotiation.cc:407] Connection header received
0928 14:46:18.702215 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.702219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.702334 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.702407 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.705805 (+ 3398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.706712 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709128 (+ 2416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.709502 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.709616 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.712495 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.712503 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.712511 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.712552 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.714744 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.723124 (+ 8380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.723129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.723131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.723266 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.723610 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.723614 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.723616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.723859 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.723969 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.724814 (+ 845us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.725028 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":49}
I20260928 14:46:18.725809 21565 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44095
I20260928 14:46:18.725912 21565 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:18.726112 21565 heartbeater.cc:511] Master 127.13.223.126:44095 requested a full tablet report, sending...
I20260928 14:46:18.726926 20325 ts_manager.cc:194] Re-registered known tserver with Master: 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:18.727628 20325 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:35995
I20260928 14:46:18.732211 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.704044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.13.223.125:43405)
0928 14:46:18.704688 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.704692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.708381 (+ 3689us) server_negotiation.cc:407] Connection header received
0928 14:46:18.708576 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.708582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.708650 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.708750 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.720504 (+ 11754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.721843 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.724455 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.725072 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.725684 (+ 612us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.728205 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.728209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.728213 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.728244 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.730037 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.730613 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.730616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.730618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.730744 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.731120 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.731125 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.731126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.731383 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.731525 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.731858 (+ 333us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.732009 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":129,"threads_started":1}
I20260928 14:46:18.731614 20325 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:18.735545 20325 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:18.738338 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: NON_VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: true } } }
I20260928 14:46:18.738610 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:18.743049 21581 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: attempting to promote NON_VOTER a801913b65ef454fa708392b7df506ea to VOTER
I20260928 14:46:18.743481 21581 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:18.744861 21113 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 LEARNER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:46:18.745425 20889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 14:46:18.747396 20937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Refusing update from remote peer cda5c6fa51af4e8c81f35f3b6f1e8828: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 14:46:18.747861 21583 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a801913b65ef454fa708392b7df506ea (127.13.223.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.748085 20889 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 14:46:18.749285 20937 raft_consensus.cc:2932] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a801913b65ef454fa708392b7df506ea (127.13.223.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.749460 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.749571 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:18.749765 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.749850 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:18.750531 21113 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, a801913b65ef454fa708392b7df506ea (127.13.223.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.751302 21591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.751442 21591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:18.751611 21583 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.751711 21583 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:18.753443 21153 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:18.753578 21153 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:18.761111 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.753762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45707 (local address 127.0.0.1:56956)
0928 14:46:18.753905 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.753922 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.754014 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.756125 (+ 2111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.756128 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.756148 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.756295 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.756304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.758098 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.758102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.759355 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.759365 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.760645 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.760649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.760746 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.760790 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.760847 (+ 57us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.760930 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":52}
I20260928 14:46:18.762058 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.754529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.13.223.67:45707)
0928 14:46:18.755297 (+ 768us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.755312 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.755339 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:46:18.755633 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.755640 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.755936 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.756484 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.756646 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.757922 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.759498 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.760486 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.761472 (+ 986us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.761671 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.761738 (+ 67us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.761868 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":116,"threads_started":1}
I20260928 14:46:18.762225 21595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.756145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41435 (local address 127.0.0.1:38412)
0928 14:46:18.756443 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.756458 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.756547 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.757447 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.757453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.757472 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.757545 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.757552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.759316 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.759319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.760360 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.760369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.761594 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.761598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.761811 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.761835 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.761923 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.762006 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":54,"thread_start_us":137,"threads_started":1}
I20260928 14:46:18.762257 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.756338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.13.223.66:41435)
0928 14:46:18.756785 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.756798 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.756850 (+ 52us) server_negotiation.cc:407] Connection header received
0928 14:46:18.756925 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.756936 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.757137 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.757316 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.757642 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.759078 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.760484 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.761097 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.761213 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.761529 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.761986 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.762115 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":100,"threads_started":1}
I20260928 14:46:18.764010 21592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.754087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41187 (local address 127.0.0.1:60344)
0928 14:46:18.754425 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.754441 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.754543 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.757280 (+ 2737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.757283 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.757303 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.757401 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.757409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.758917 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.758921 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.760309 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.760322 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.761594 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.761598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.763653 (+ 2055us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.763682 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.763771 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.763858 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":63,"thread_start_us":179,"threads_started":1}
I20260928 14:46:18.764154 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.754992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60344 (local address 127.13.223.65:41187)
0928 14:46:18.756815 (+ 1823us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.756822 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.756848 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:46:18.756927 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.756932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.757135 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.757384 (+ 249us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.757551 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.758766 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.760497 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.761460 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.761592 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.761718 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.763855 (+ 2137us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.763987 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1682,"thread_start_us":75,"threads_started":1}
I20260928 14:46:18.765483 21498 tablet_service.cc:1467] Processing CreateTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 (DEFAULT_TABLE table=test-table [id=8b13dbdb1c2e4551ada847079f02ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:18.765522 21361 tablet_service.cc:1467] Processing CreateTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 (DEFAULT_TABLE table=test-table [id=8b13dbdb1c2e4551ada847079f02ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:18.766646 21498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3254152a45554fd78e72c59a5c0d16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:18.766748 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3254152a45554fd78e72c59a5c0d16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:18.767704 21224 tablet_service.cc:1467] Processing CreateTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 (DEFAULT_TABLE table=test-table [id=8b13dbdb1c2e4551ada847079f02ffc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:18.769059 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3254152a45554fd78e72c59a5c0d16e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:18.774175 21597 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Bootstrap starting.
I20260928 14:46:18.775717 21598 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Bootstrap starting.
I20260928 14:46:18.776782 21597 tablet_bootstrap.cc:654] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:18.777746 21597 log.cc:824] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:18.778162 21598 tablet_bootstrap.cc:654] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:18.778541 21599 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Bootstrap starting.
I20260928 14:46:18.779079 21598 log.cc:824] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:18.780877 21599 tablet_bootstrap.cc:654] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:18.780962 21597 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: No bootstrap required, opened a new log
I20260928 14:46:18.781174 21597 ts_tablet_manager.cc:1397] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:46:18.781322 21598 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: No bootstrap required, opened a new log
I20260928 14:46:18.781541 21598 ts_tablet_manager.cc:1397] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:46:18.781847 21599 log.cc:824] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:18.783674 21599 tablet_bootstrap.cc:492] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: No bootstrap required, opened a new log
I20260928 14:46:18.783902 21599 ts_tablet_manager.cc:1397] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:46:18.786298 21597 raft_consensus.cc:348] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.786298 21598 raft_consensus.cc:348] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.786628 21597 raft_consensus.cc:374] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:18.786628 21598 raft_consensus.cc:374] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:18.786690 21597 raft_consensus.cc:729] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8612fb089f409d9faffefe7c9cdec4, State: Initialized, Role: FOLLOWER
I20260928 14:46:18.786688 21598 raft_consensus.cc:729] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504a5636894b4f94bbae983fd2717e57, State: Initialized, Role: FOLLOWER
I20260928 14:46:18.787218 21597 consensus_queue.cc:260] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.787218 21598 consensus_queue.cc:260] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.788308 21428 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
I20260928 14:46:18.788352 21597 ts_tablet_manager.cc:1428] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 14:46:18.788352 21598 ts_tablet_manager.cc:1428] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:18.789729 21565 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
I20260928 14:46:18.789642 21599 raft_consensus.cc:348] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.790017 21599 raft_consensus.cc:374] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:18.790077 21599 raft_consensus.cc:729] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c83b31373ec423c8f4524befc90bfc4, State: Initialized, Role: FOLLOWER
I20260928 14:46:18.791190 21599 consensus_queue.cc:260] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.794041 21291 heartbeater.cc:503] Master 127.13.223.126:44095 was elected leader, sending a full tablet report...
I20260928 14:46:18.794879 21599 ts_tablet_manager.cc:1428] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260928 14:46:18.808060 21603 raft_consensus.cc:482] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:18.808254 21603 raft_consensus.cc:504] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.809405 21603 leader_election.cc:288] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187), 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
I20260928 14:46:18.813925 21594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.809807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:34359 (local address 127.13.223.65:41187)
0928 14:46:18.809970 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.809976 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.809999 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:46:18.810061 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.810066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.810122 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.810204 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.810586 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.811313 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.812788 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.813205 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.813309 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.813380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.813665 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.813764 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:46:18.813928 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.809676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41435 (local address 127.13.223.67:49041)
0928 14:46:18.810036 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.810049 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.810128 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.810411 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.810413 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.810433 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.810617 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.810625 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.811562 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.811565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.812598 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.812609 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.813465 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.813468 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.813551 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.813569 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.813678 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.813760 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":212}
I20260928 14:46:18.813966 21596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.809815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:49041 (local address 127.13.223.66:41435)
0928 14:46:18.809972 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.809980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.810108 (+ 128us) server_negotiation.cc:407] Connection header received
0928 14:46:18.810246 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.810249 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.810302 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.810392 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:18.810736 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.811418 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.812761 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.813288 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.813427 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.813481 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.813752 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.813844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 14:46:18.813935 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.809659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41187 (local address 127.13.223.67:34359)
0928 14:46:18.809814 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:18.809834 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:18.809948 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:18.810235 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:18.810238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:18.810260 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:18.810424 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.810433 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.811432 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.811436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.812623 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:18.812634 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.813351 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.813359 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:18.813484 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.813531 (+ 47us) client_negotiation.cc:769] Sending connection context
0928 14:46:18.813615 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:46:18.813752 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":24}
I20260928 14:46:18.814807 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3254152a45554fd78e72c59a5c0d16e6" candidate_uuid: "4b8612fb089f409d9faffefe7c9cdec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504a5636894b4f94bbae983fd2717e57" is_pre_election: true
I20260928 14:46:18.815176 21381 raft_consensus.cc:2454] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8612fb089f409d9faffefe7c9cdec4 in term 0.
I20260928 14:46:18.815264 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3254152a45554fd78e72c59a5c0d16e6" candidate_uuid: "4b8612fb089f409d9faffefe7c9cdec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c83b31373ec423c8f4524befc90bfc4" is_pre_election: true
I20260928 14:46:18.815667 21244 raft_consensus.cc:2454] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b8612fb089f409d9faffefe7c9cdec4 in term 0.
I20260928 14:46:18.815848 21452 leader_election.cc:302] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8612fb089f409d9faffefe7c9cdec4, 504a5636894b4f94bbae983fd2717e57; no voters:
I20260928 14:46:18.816195 21603 raft_consensus.cc:2781] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:18.816283 21603 raft_consensus.cc:482] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:18.816342 21603 raft_consensus.cc:3037] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:18.817528 21603 raft_consensus.cc:504] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.817922 21603 leader_election.cc:288] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [CANDIDATE]: Term 1 election: Requested vote from peers 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187), 504a5636894b4f94bbae983fd2717e57 (127.13.223.66:41435)
I20260928 14:46:18.818324 21381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3254152a45554fd78e72c59a5c0d16e6" candidate_uuid: "4b8612fb089f409d9faffefe7c9cdec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "504a5636894b4f94bbae983fd2717e57"
I20260928 14:46:18.818501 21381 raft_consensus.cc:3037] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:18.818503 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3254152a45554fd78e72c59a5c0d16e6" candidate_uuid: "4b8612fb089f409d9faffefe7c9cdec4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c83b31373ec423c8f4524befc90bfc4"
I20260928 14:46:18.818681 21244 raft_consensus.cc:3037] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:18.819871 21244 raft_consensus.cc:2454] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8612fb089f409d9faffefe7c9cdec4 in term 1.
I20260928 14:46:18.820078 21381 raft_consensus.cc:2454] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b8612fb089f409d9faffefe7c9cdec4 in term 1.
I20260928 14:46:18.820282 21453 leader_election.cc:302] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c83b31373ec423c8f4524befc90bfc4, 4b8612fb089f409d9faffefe7c9cdec4; no voters:
I20260928 14:46:18.820585 21603 raft_consensus.cc:2781] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:18.820847 21603 raft_consensus.cc:686] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 LEADER]: Becoming Leader. State: Replica: 4b8612fb089f409d9faffefe7c9cdec4, State: Running, Role: LEADER
I20260928 14:46:18.821202 21603 consensus_queue.cc:237] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:18.824977 20325 catalog_manager.cc:5495] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "4b8612fb089f409d9faffefe7c9cdec4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:18.846590 21593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.842395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56958 (local address 127.13.223.67:45707)
0928 14:46:18.842615 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.842624 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.842642 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:18.842741 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.842743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.842802 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.842942 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:18.843220 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.843797 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.844824 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.845078 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.845207 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.845588 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:18.846081 (+ 493us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:18.846227 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.846296 (+ 69us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.846363 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:46:18.853529 21244 raft_consensus.cc:1261] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Refusing update from remote peer 4b8612fb089f409d9faffefe7c9cdec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:18.853529 21381 raft_consensus.cc:1261] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Refusing update from remote peer 4b8612fb089f409d9faffefe7c9cdec4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:18.854457 21606 consensus_queue.cc:1035] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:18.854769 21603 consensus_queue.cc:1035] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:18.865053 21609 mvcc.cc:205] Tried to move back new op lower bound from 7334325366175318016 to 7334325366054522880. Current Snapshot: MvccSnapshot[applied={T|T < 7334325366175318016}]
I20260928 14:46:18.869280 21610 mvcc.cc:205] Tried to move back new op lower bound from 7334325366175318016 to 7334325366054522880. Current Snapshot: MvccSnapshot[applied={T|T < 7334325366175318016}]
I20260928 14:46:18.871981 21611 mvcc.cc:205] Tried to move back new op lower bound from 7334325366175318016 to 7334325366054522880. Current Snapshot: MvccSnapshot[applied={T|T < 7334325366175318016}]
I20260928 14:46:18.880033 20324 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:18.880255 20324 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:18.885842 20324 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cda5c6fa51af4e8c81f35f3b6f1e8828: Sending DeleteTablet for 3 replicas of tablet 3254152a45554fd78e72c59a5c0d16e6
I20260928 14:46:18.886766 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:18 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:60344
I20260928 14:46:18.886999 21498 tablet_service.cc:1514] Processing DeleteTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:18 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:56956
I20260928 14:46:18.887421 21619 tablet_replica.cc:330] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: stopping tablet replica
I20260928 14:46:18.887601 21361 tablet_service.cc:1514] Processing DeleteTablet for tablet 3254152a45554fd78e72c59a5c0d16e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:18 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38412
I20260928 14:46:18.887815 21619 raft_consensus.cc:2229] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:18.888147 21621 tablet_replica.cc:330] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: stopping tablet replica
I20260928 14:46:18.888204 21619 raft_consensus.cc:2258] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:18.889008 21620 tablet_replica.cc:330] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: stopping tablet replica
I20260928 14:46:18.889328 21620 raft_consensus.cc:2229] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:18.889767 21620 raft_consensus.cc:2258] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:18.889940 21619 ts_tablet_manager.cc:1905] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:18.890046 21621 raft_consensus.cc:2229] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:18.890403 21621 raft_consensus.cc:2258] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:18.890810 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20293
I20260928 14:46:18.893600 21620 ts_tablet_manager.cc:1905] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:18.893627 21619 ts_tablet_manager.cc:1918] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:18.893730 21619 log.cc:1196] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/3254152a45554fd78e72c59a5c0d16e6
I20260928 14:46:18.894076 21619 ts_tablet_manager.cc:1939] T 3254152a45554fd78e72c59a5c0d16e6 P 0c83b31373ec423c8f4524befc90bfc4: Deleting consensus metadata
I20260928 14:46:18.897670 21620 ts_tablet_manager.cc:1918] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:18.897792 21620 log.cc:1196] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/3254152a45554fd78e72c59a5c0d16e6
I20260928 14:46:18.898171 21620 ts_tablet_manager.cc:1939] T 3254152a45554fd78e72c59a5c0d16e6 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting consensus metadata
I20260928 14:46:18.899502 21621 ts_tablet_manager.cc:1905] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:46:18.900230 21180 connection.cc:535] server connection from 127.0.0.1:60344 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:46:18.902683 21621 ts_tablet_manager.cc:1918] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:18.902796 21621 log.cc:1196] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/3254152a45554fd78e72c59a5c0d16e6
I20260928 14:46:18.903177 21621 ts_tablet_manager.cc:1939] T 3254152a45554fd78e72c59a5c0d16e6 P 504a5636894b4f94bbae983fd2717e57: Deleting consensus metadata
W20260928 14:46:18.904402 21454 connection.cc:441] server connection from 127.0.0.1:56956 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56956 (request call id 1) could send its response
W20260928 14:46:18.905468 21317 connection.cc:441] server connection from 127.0.0.1:38412 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38412 (request call id 1) could send its response
I20260928 14:46:18.911254 21578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.888015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.13.223.125:43405)
0928 14:46:18.888137 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.888141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.898972 (+ 10831us) server_negotiation.cc:407] Connection header received
0928 14:46:18.899236 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.899242 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.899319 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.899388 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.900668 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.901595 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.903632 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.904045 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.904147 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.906513 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.906518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.906521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.906590 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.908817 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.909555 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.909562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.909567 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.909673 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.910013 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.910017 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.910019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.910258 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.910406 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.910900 (+ 494us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.911060 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:46:18.924232 21050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.906344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51504 (local address 127.13.223.125:43405)
0928 14:46:18.906486 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.906491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.907201 (+ 710us) server_negotiation.cc:407] Connection header received
0928 14:46:18.907357 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.907360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.907407 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.907471 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.908500 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.909354 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.911379 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.911734 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.911807 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.913723 (+ 1916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.913728 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.913732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.913764 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.921309 (+ 7545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.922024 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.922031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.922035 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.922189 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.922914 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.922922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.922925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.923236 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.923367 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.923869 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.924040 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:18.924647 21294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.905565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.13.223.124:44517)
0928 14:46:18.905812 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.905819 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.906884 (+ 1065us) server_negotiation.cc:407] Connection header received
0928 14:46:18.907023 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.907029 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.907117 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.907213 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.908509 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.909448 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.910919 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.911417 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.911512 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.920494 (+ 8982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.920499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.920502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.920539 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.922765 (+ 2226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.923287 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.923291 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.923293 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.923384 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.923695 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.923698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.923700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.923912 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.924021 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.924352 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.924493 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260928 14:46:18.925482 21433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:18.888666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.13.223.124:44517)
0928 14:46:18.888785 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:18.888791 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:18.912662 (+ 23871us) server_negotiation.cc:407] Connection header received
0928 14:46:18.912935 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:18.912939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:18.912998 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:18.913062 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:18.914377 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.915248 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.916160 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:18.916522 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:18.916628 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:18.920640 (+ 4012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:18.920647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:18.920653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:18.920694 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:18.923024 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.923583 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.923587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.923589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.923685 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:18.924042 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:18.924046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:18.924047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:18.924268 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:18.924377 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:18.925210 (+ 833us) server_negotiation.cc:299] Negotiation successful
0928 14:46:18.925330 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:46:19.870929 21630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:19.870240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:52263)
0928 14:46:19.870620 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:19.870768 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":104,"threads_started":1}
W20260928 14:46:19.871634 21565 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:19.871778 21631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:19.871323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:46457)
0928 14:46:19.871580 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:19.871680 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":85,"threads_started":1}
W20260928 14:46:19.872722 21291 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:19.873319 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:19.872852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:39679)
0928 14:46:19.873157 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:19.873232 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":130,"threads_started":1}
W20260928 14:46:19.873736 21428 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44095 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.386041 21633 raft_consensus.cc:482] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cda5c6fa51af4e8c81f35f3b6f1e8828)
I20260928 14:46:20.386138 21634 raft_consensus.cc:482] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cda5c6fa51af4e8c81f35f3b6f1e8828)
I20260928 14:46:20.386291 21633 raft_consensus.cc:504] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:20.386399 21634 raft_consensus.cc:504] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:20.387475 21634 leader_election.cc:288] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125:43405)
I20260928 14:46:20.387926 21633 leader_election.cc:288] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), a801913b65ef454fa708392b7df506ea (127.13.223.124:44517)
I20260928 14:46:20.388177 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.387660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42982)
0928 14:46:20.387968 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.388057 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":69,"threads_started":1}
W20260928 14:46:20.388430 20914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.388566 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.387639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42980)
0928 14:46:20.388405 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.388478 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":70,"threads_started":1}
W20260928 14:46:20.388731 21097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.389487 21636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.389264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:42998)
0928 14:46:20.389371 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.389423 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 14:46:20.389595 20914 leader_election.cc:334] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.389860 21635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.389377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:43014)
0928 14:46:20.389469 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.389522 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:46:20.390043 21097 leader_election.cc:334] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.392813 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.388024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:43405 (local address 127.0.0.1:51520)
0928 14:46:20.388704 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.388718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:20.388834 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:20.389244 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:20.389247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:20.389263 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:20.389402 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:20.389411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.390612 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.390618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:20.391468 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:20.391477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.392264 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.392268 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:20.392440 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:20.392500 (+ 60us) client_negotiation.cc:769] Sending connection context
0928 14:46:20.392579 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 14:46:20.392669 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":27,"thread_start_us":78,"threads_started":1}
I20260928 14:46:20.392884 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.387711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:44517 (local address 127.0.0.1:48042)
0928 14:46:20.388227 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.388243 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:20.388365 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:20.389666 (+ 1301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:20.389669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:20.389688 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:20.389762 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:20.389768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.390671 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.390673 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:20.391521 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:20.391560 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.392415 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.392418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:20.392495 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:20.392514 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:46:20.392637 (+ 123us) client_negotiation.cc:240] Negotiation successful
0928 14:46:20.392734 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"mutex_wait_us":25,"thread_start_us":83,"threads_started":1}
I20260928 14:46:20.392940 21639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.388075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51520 (local address 127.13.223.125:43405)
0928 14:46:20.388463 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:20.388471 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:20.388811 (+ 340us) server_negotiation.cc:407] Connection header received
0928 14:46:20.388950 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:20.388956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:20.389025 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:20.389117 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:20.389627 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.390462 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.391603 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.392083 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.392188 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:20.392232 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:20.392622 (+ 390us) server_negotiation.cc:299] Negotiation successful
0928 14:46:20.392731 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":308,"thread_start_us":61,"threads_started":1}
I20260928 14:46:20.392813 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.388226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48042 (local address 127.13.223.124:44517)
0928 14:46:20.389126 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:20.389131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:20.389144 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:46:20.389190 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:20.389193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:20.389243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:20.389553 (+ 310us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:20.389884 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.390557 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.391684 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:20.392265 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:20.392453 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:20.392553 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:20.392649 (+ 96us) server_negotiation.cc:299] Negotiation successful
0928 14:46:20.392704 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":304,"thread_start_us":57,"threads_started":1}
I20260928 14:46:20.393610 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a801913b65ef454fa708392b7df506ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" is_pre_election: true
I20260928 14:46:20.393610 21113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a801913b65ef454fa708392b7df506ea" is_pre_election: true
I20260928 14:46:20.393842 21113 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d12c09dbbaf54e06b2cec5d0403c7d5f in term 1.
I20260928 14:46:20.393842 20937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a801913b65ef454fa708392b7df506ea in term 1.
I20260928 14:46:20.394213 21097 leader_election.cc:302] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a801913b65ef454fa708392b7df506ea, d12c09dbbaf54e06b2cec5d0403c7d5f; no voters: cda5c6fa51af4e8c81f35f3b6f1e8828
I20260928 14:46:20.394215 20915 leader_election.cc:302] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a801913b65ef454fa708392b7df506ea, d12c09dbbaf54e06b2cec5d0403c7d5f; no voters: cda5c6fa51af4e8c81f35f3b6f1e8828
I20260928 14:46:20.394452 21634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:46:20.394529 21634 raft_consensus.cc:482] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Starting leader election (detected failure of leader cda5c6fa51af4e8c81f35f3b6f1e8828)
I20260928 14:46:20.394613 21634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:46:20.394634 21633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 14:46:20.394752 21633 raft_consensus.cc:482] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Starting leader election (detected failure of leader cda5c6fa51af4e8c81f35f3b6f1e8828)
I20260928 14:46:20.394832 21633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:46:20.395852 21633 raft_consensus.cc:504] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:20.395852 21634 raft_consensus.cc:504] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:20.396260 21634 leader_election.cc:288] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 election: Requested vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125:43405)
I20260928 14:46:20.396299 21633 leader_election.cc:288] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 election: Requested vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), a801913b65ef454fa708392b7df506ea (127.13.223.124:44517)
I20260928 14:46:20.396683 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.396412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:43018)
0928 14:46:20.396548 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.396609 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:46:20.396716 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.396466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:43024)
0928 14:46:20.396589 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.396645 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 14:46:20.396898 21113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a801913b65ef454fa708392b7df506ea"
I20260928 14:46:20.397080 21113 raft_consensus.cc:2379] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d12c09dbbaf54e06b2cec5d0403c7d5f in current term 2: Already voted for candidate a801913b65ef454fa708392b7df506ea in this term.
I20260928 14:46:20.397287 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a801913b65ef454fa708392b7df506ea" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
I20260928 14:46:20.397387 21637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.397175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:43036)
0928 14:46:20.397263 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.397314 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260928 14:46:20.397464 20937 raft_consensus.cc:2379] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate a801913b65ef454fa708392b7df506ea in current term 2: Already voted for candidate d12c09dbbaf54e06b2cec5d0403c7d5f in this term.
W20260928 14:46:20.397533 20914 leader_election.cc:334] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.397852 20915 leader_election.cc:302] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d12c09dbbaf54e06b2cec5d0403c7d5f; no voters: a801913b65ef454fa708392b7df506ea, cda5c6fa51af4e8c81f35f3b6f1e8828
I20260928 14:46:20.398054 21633 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:46:20.398315 21638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.398049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:43044)
0928 14:46:20.398202 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.398258 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:46:20.398433 21097 leader_election.cc:334] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:20.398531 21097 leader_election.cc:302] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a801913b65ef454fa708392b7df506ea; no voters: cda5c6fa51af4e8c81f35f3b6f1e8828, d12c09dbbaf54e06b2cec5d0403c7d5f
I20260928 14:46:20.398717 21634 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 14:46:20.872958 21643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.872445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:58975)
0928 14:46:20.872747 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.872823 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260928 14:46:20.873868 21644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.873360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:36931)
0928 14:46:20.873649 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.873723 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":71,"threads_started":1}
I20260928 14:46:20.875011 21645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:20.874379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:55981)
0928 14:46:20.874873 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:20.874928 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":92,"threads_started":1}
I20260928 14:46:21.874672 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:21.874078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:46309)
0928 14:46:21.874444 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:21.874522 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":84,"threads_started":1}
I20260928 14:46:21.875507 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:21.875029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:47623)
0928 14:46:21.875336 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:21.875403 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":70,"threads_started":1}
I20260928 14:46:21.876080 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:21.875674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:41765)
0928 14:46:21.875930 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:21.875993 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":78,"threads_started":1}
I20260928 14:46:22.176252 21649 raft_consensus.cc:482] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:22.176407 21649 raft_consensus.cc:504] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:22.176883 21649 leader_election.cc:288] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125:43405)
I20260928 14:46:22.177616 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.177116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33004)
0928 14:46:22.177409 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.177489 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":140,"threads_started":1}
I20260928 14:46:22.177806 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a801913b65ef454fa708392b7df506ea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" is_pre_election: true
I20260928 14:46:22.178005 20937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a801913b65ef454fa708392b7df506ea in term 2.
I20260928 14:46:22.178623 21097 leader_election.cc:302] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a801913b65ef454fa708392b7df506ea, d12c09dbbaf54e06b2cec5d0403c7d5f; no voters:
I20260928 14:46:22.178820 21649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 14:46:22.178985 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.178425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33010)
0928 14:46:22.178659 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.178756 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 14:46:22.179010 21649 raft_consensus.cc:482] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:22.179065 21649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 2 FOLLOWER]: Advancing to term 3
W20260928 14:46:22.179126 21097 leader_election.cc:334] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:22.180466 21649 raft_consensus.cc:504] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:22.180861 21649 leader_election.cc:288] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 election: Requested vote from peers cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095), d12c09dbbaf54e06b2cec5d0403c7d5f (127.13.223.125:43405)
I20260928 14:46:22.181430 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.181092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33014)
0928 14:46:22.181266 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.181327 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 14:46:22.181646 20937 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a801913b65ef454fa708392b7df506ea" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f"
I20260928 14:46:22.181802 20937 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:46:22.182173 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.181951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33022)
0928 14:46:22.182065 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.182119 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:46:22.182276 21097 leader_election.cc:334] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
I20260928 14:46:22.183056 20937 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a801913b65ef454fa708392b7df506ea in term 3.
I20260928 14:46:22.183432 21097 leader_election.cc:302] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a801913b65ef454fa708392b7df506ea, d12c09dbbaf54e06b2cec5d0403c7d5f; no voters: cda5c6fa51af4e8c81f35f3b6f1e8828
I20260928 14:46:22.183642 21649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:46:22.183905 21649 raft_consensus.cc:686] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [term 3 LEADER]: Becoming Leader. State: Replica: a801913b65ef454fa708392b7df506ea, State: Running, Role: LEADER
I20260928 14:46:22.184186 21649 consensus_queue.cc:237] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } }
I20260928 14:46:22.185855 21652 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader a801913b65ef454fa708392b7df506ea. Latest consensus state: current_term: 3 leader_uuid: "a801913b65ef454fa708392b7df506ea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:22.186101 21652 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:22.186429 21654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:22.189147 21654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:22.189590 21654 catalog_manager.cc:1259] Loaded cluster ID: dd07be5537b5430daf44285a88ecf056
I20260928 14:46:22.189646 21654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:22.190207 21654 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:22.190682 21654 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Loaded TSK: 0
I20260928 14:46:22.191396 21654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:22.252703 21103 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:22.256424 21103 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:22.260284 20937 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [term 3 FOLLOWER]: Refusing update from remote peer a801913b65ef454fa708392b7df506ea: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 14:46:22.260344 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.259984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33050)
0928 14:46:22.260146 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.260242 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260928 14:46:22.260864 21649 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 14:46:22.261265 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.260978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.0.0.1:33060)
0928 14:46:22.261149 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.261206 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260928 14:46:22.261559 21097 consensus_peers.cc:487] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea -> Peer cda5c6fa51af4e8c81f35f3b6f1e8828 (127.13.223.126:44095): Couldn't send request to peer cda5c6fa51af4e8c81f35f3b6f1e8828. Status: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 14:46:22.262605 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a801913b65ef454fa708392b7df506ea. Latest consensus state: current_term: 3 leader_uuid: "a801913b65ef454fa708392b7df506ea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:22.262734 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:22.264369 21649 sys_catalog.cc:455] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "a801913b65ef454fa708392b7df506ea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:22.264509 21649 sys_catalog.cc:458] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:22.264801 21657 mvcc.cc:205] Tried to move back new op lower bound from 7334325380132323328 to 7334325379829301248. Current Snapshot: MvccSnapshot[applied={T|T < 7334325380132323328}]
I20260928 14:46:22.265040 21656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "a801913b65ef454fa708392b7df506ea" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cda5c6fa51af4e8c81f35f3b6f1e8828" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44095 } } peers { permanent_uuid: "d12c09dbbaf54e06b2cec5d0403c7d5f" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 43405 } attrs { promote: false } } peers { permanent_uuid: "a801913b65ef454fa708392b7df506ea" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 44517 } attrs { promote: false } } }
I20260928 14:46:22.267275 21656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d12c09dbbaf54e06b2cec5d0403c7d5f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:46:22.277925 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.273019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45707 (local address 127.0.0.1:36728)
0928 14:46:22.273203 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.273219 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.273352 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.273799 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:22.273802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:22.273820 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:22.273946 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.273953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.275109 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.275113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.276622 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.276631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.277465 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.277469 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.277579 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.277601 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:46:22.277672 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:46:22.277762 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":279}
I20260928 14:46:22.278369 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.273249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.13.223.67:45707)
0928 14:46:22.273520 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.273525 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.273540 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:46:22.273596 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.273599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.273659 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.274066 (+ 407us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:22.274198 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.274974 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.276761 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.277314 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.277450 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.277553 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.278127 (+ 574us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.278241 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":82,"threads_started":1}
I20260928 14:46:22.278963 21498 tablet_service.cc:1467] Processing CreateTablet for tablet d012cb7b506e4caaa14e05965abfefe5 (DEFAULT_TABLE table=test-table [id=ab9b3458ce074b15833228a80401df3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:22.279263 21498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d012cb7b506e4caaa14e05965abfefe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:22.280143 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.273960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.13.223.65:41187)
0928 14:46:22.275605 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.275609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.275624 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:46:22.275677 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.275680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.275727 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.276081 (+ 354us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:22.276187 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.276917 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.279019 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.279611 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.279705 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.279757 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.279974 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.280035 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1553,"thread_start_us":76,"threads_started":1}
I20260928 14:46:22.280167 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.273579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41187 (local address 127.0.0.1:51484)
0928 14:46:22.273924 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.273940 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.274029 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.275860 (+ 1831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:22.275863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:22.275879 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:22.275974 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.275980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.277042 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.277046 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.278840 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.278850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.279751 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.279755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.279846 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.279871 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:46:22.279955 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:46:22.280028 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":116,"threads_started":1}
I20260928 14:46:22.280810 21224 tablet_service.cc:1467] Processing CreateTablet for tablet d012cb7b506e4caaa14e05965abfefe5 (DEFAULT_TABLE table=test-table [id=ab9b3458ce074b15833228a80401df3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:22.281134 21224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d012cb7b506e4caaa14e05965abfefe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:22.282015 21667 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Bootstrap starting.
I20260928 14:46:22.282656 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.274272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41435 (local address 127.0.0.1:57970)
0928 14:46:22.274505 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.274520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.274632 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.275045 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:22.275048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:22.275064 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:22.275410 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.275423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.276320 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.276326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.277314 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.277323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.278368 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.278372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.278463 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.278486 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:46:22.278585 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:46:22.282454 (+ 3869us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":1,"thread_start_us":93,"threads_started":1}
I20260928 14:46:22.283120 21667 tablet_bootstrap.cc:654] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:22.283429 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.274427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57970 (local address 127.13.223.66:41435)
0928 14:46:22.274698 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.274703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.274720 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:22.274772 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.274775 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.274828 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.274923 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:22.275559 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.276105 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.277485 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.278148 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.278250 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.278313 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.282693 (+ 4380us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.283178 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":66,"threads_started":1}
I20260928 14:46:22.284024 21361 tablet_service.cc:1467] Processing CreateTablet for tablet d012cb7b506e4caaa14e05965abfefe5 (DEFAULT_TABLE table=test-table [id=ab9b3458ce074b15833228a80401df3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:22.284348 21361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d012cb7b506e4caaa14e05965abfefe5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:22.284797 21667 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: No bootstrap required, opened a new log
I20260928 14:46:22.284919 21667 ts_tablet_manager.cc:1397] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:22.285429 21667 raft_consensus.cc:348] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.285583 21667 raft_consensus.cc:374] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:22.285636 21667 raft_consensus.cc:729] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b8612fb089f409d9faffefe7c9cdec4, State: Initialized, Role: FOLLOWER
I20260928 14:46:22.285779 21667 consensus_queue.cc:260] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.286266 21667 ts_tablet_manager.cc:1428] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:46:22.287202 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.286804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:35391)
0928 14:46:22.287058 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.287124 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260928 14:46:22.287343 21670 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: Bootstrap starting.
I20260928 14:46:22.288618 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.288343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:40977)
0928 14:46:22.288482 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.288544 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:46:22.288700 21670 tablet_bootstrap.cc:654] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:22.289875 21670 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: No bootstrap required, opened a new log
I20260928 14:46:22.290009 21670 ts_tablet_manager.cc:1397] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:22.290310 21564 heartbeater.cc:503] Master 127.13.223.124:44517 was elected leader, sending a full tablet report...
I20260928 14:46:22.290522 21670 raft_consensus.cc:348] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.290719 21670 raft_consensus.cc:374] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:22.290769 21670 raft_consensus.cc:729] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 504a5636894b4f94bbae983fd2717e57, State: Initialized, Role: FOLLOWER
I20260928 14:46:22.290917 21670 consensus_queue.cc:260] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.291065 21671 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: Bootstrap starting.
I20260928 14:46:22.292192 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.291650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:38631)
0928 14:46:22.292022 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.292096 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260928 14:46:22.293062 21670 ts_tablet_manager.cc:1428] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:46:22.293246 21427 heartbeater.cc:503] Master 127.13.223.124:44517 was elected leader, sending a full tablet report...
I20260928 14:46:22.293902 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.293495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:59035)
0928 14:46:22.293744 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.293803 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260928 14:46:22.294757 21671 tablet_bootstrap.cc:654] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:22.296161 21671 tablet_bootstrap.cc:492] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: No bootstrap required, opened a new log
I20260928 14:46:22.296413 21671 ts_tablet_manager.cc:1397] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260928 14:46:22.296969 21671 raft_consensus.cc:348] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.297117 21671 raft_consensus.cc:374] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:22.297151 21671 raft_consensus.cc:729] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c83b31373ec423c8f4524befc90bfc4, State: Initialized, Role: FOLLOWER
I20260928 14:46:22.297318 21671 consensus_queue.cc:260] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.297760 21671 ts_tablet_manager.cc:1428] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20260928 14:46:22.298709 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.298306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:33015)
0928 14:46:22.298484 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.298617 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:46:22.299887 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.299592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:53689)
0928 14:46:22.299719 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.299794 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260928 14:46:22.300141 21290 heartbeater.cc:503] Master 127.13.223.124:44517 was elected leader, sending a full tablet report...
I20260928 14:46:22.363227 21673 raft_consensus.cc:482] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:22.363399 21673 raft_consensus.cc:504] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.364766 21673 leader_election.cc:288] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187), 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:22.371157 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.365066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45707 (local address 127.13.223.66:44591)
0928 14:46:22.365296 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.365314 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.365447 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.365785 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:22.365789 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:22.365807 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:22.365992 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.366000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.367263 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.367268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.369218 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.369237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.370364 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.370369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.370639 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.370745 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 14:46:22.370844 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:46:22.370967 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":417}
I20260928 14:46:22.371218 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.365224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:56157 (local address 127.13.223.65:41187)
0928 14:46:22.365381 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.365386 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.365634 (+ 248us) server_negotiation.cc:407] Connection header received
0928 14:46:22.365739 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.365742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.365788 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.365858 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:22.366194 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.367077 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.369457 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.370136 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.370330 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.370395 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.370980 (+ 585us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.371085 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:46:22.371218 21662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.365181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44591 (local address 127.13.223.67:45707)
0928 14:46:22.365320 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.365326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.365415 (+ 89us) server_negotiation.cc:407] Connection header received
0928 14:46:22.365574 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.365577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.365662 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.365748 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:22.366182 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.367117 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.369457 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.370195 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.370330 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.370395 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.370881 (+ 486us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.371087 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:46:22.371157 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.365104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41187 (local address 127.13.223.66:56157)
0928 14:46:22.365518 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.365534 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.365633 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.365934 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:22.365940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:22.365962 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:22.366055 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.366064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.367263 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.367268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.369218 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:22.369237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.370364 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.370369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:22.370660 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.370745 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 14:46:22.370946 (+ 201us) client_negotiation.cc:240] Negotiation successful
0928 14:46:22.371006 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":271,"thread_start_us":127,"threads_started":1}
I20260928 14:46:22.371809 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d012cb7b506e4caaa14e05965abfefe5" candidate_uuid: "504a5636894b4f94bbae983fd2717e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c83b31373ec423c8f4524befc90bfc4" is_pre_election: true
I20260928 14:46:22.371892 21518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d012cb7b506e4caaa14e05965abfefe5" candidate_uuid: "504a5636894b4f94bbae983fd2717e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8612fb089f409d9faffefe7c9cdec4" is_pre_election: true
I20260928 14:46:22.372030 21244 raft_consensus.cc:2454] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504a5636894b4f94bbae983fd2717e57 in term 0.
I20260928 14:46:22.372205 21518 raft_consensus.cc:2454] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 504a5636894b4f94bbae983fd2717e57 in term 0.
I20260928 14:46:22.372622 21316 leader_election.cc:302] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c83b31373ec423c8f4524befc90bfc4, 504a5636894b4f94bbae983fd2717e57; no voters:
I20260928 14:46:22.373023 21673 raft_consensus.cc:2781] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:22.373086 21673 raft_consensus.cc:482] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:22.373114 21673 raft_consensus.cc:3037] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:22.374249 21673 raft_consensus.cc:504] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.374766 21673 leader_election.cc:288] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [CANDIDATE]: Term 1 election: Requested vote from peers 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187), 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707)
I20260928 14:46:22.375200 21518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d012cb7b506e4caaa14e05965abfefe5" candidate_uuid: "504a5636894b4f94bbae983fd2717e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b8612fb089f409d9faffefe7c9cdec4"
I20260928 14:46:22.375223 21244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d012cb7b506e4caaa14e05965abfefe5" candidate_uuid: "504a5636894b4f94bbae983fd2717e57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0c83b31373ec423c8f4524befc90bfc4"
I20260928 14:46:22.375360 21244 raft_consensus.cc:3037] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:22.375356 21518 raft_consensus.cc:3037] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:22.376652 21518 raft_consensus.cc:2454] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504a5636894b4f94bbae983fd2717e57 in term 1.
I20260928 14:46:22.376652 21244 raft_consensus.cc:2454] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 504a5636894b4f94bbae983fd2717e57 in term 1.
I20260928 14:46:22.377102 21317 leader_election.cc:302] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b8612fb089f409d9faffefe7c9cdec4, 504a5636894b4f94bbae983fd2717e57; no voters:
I20260928 14:46:22.377370 21673 raft_consensus.cc:2781] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:22.377645 21673 raft_consensus.cc:686] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 1 LEADER]: Becoming Leader. State: Replica: 504a5636894b4f94bbae983fd2717e57, State: Running, Role: LEADER
I20260928 14:46:22.377959 21673 consensus_queue.cc:237] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } }
I20260928 14:46:22.378486 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.378225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:50055)
0928 14:46:22.378332 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.378398 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 14:46:22.381024 21105 catalog_manager.cc:5495] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 504a5636894b4f94bbae983fd2717e57 (127.13.223.66). New cstate: current_term: 1 leader_uuid: "504a5636894b4f94bbae983fd2717e57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "504a5636894b4f94bbae983fd2717e57" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41435 } health_report { overall_health: HEALTHY } } }
I20260928 14:46:22.410137 21666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.405703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57974 (local address 127.13.223.66:41435)
0928 14:46:22.405850 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:22.405854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:22.405870 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:22.406057 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:22.406060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:22.406123 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:22.406220 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:22.406572 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.407210 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.408352 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:22.408608 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:22.408699 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:22.408990 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:22.409561 (+ 571us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:22.409659 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:22.409794 (+ 135us) server_negotiation.cc:299] Negotiation successful
0928 14:46:22.409857 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:46:22.413704 21244 raft_consensus.cc:1261] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Refusing update from remote peer 504a5636894b4f94bbae983fd2717e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:22.414036 21518 raft_consensus.cc:1261] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Refusing update from remote peer 504a5636894b4f94bbae983fd2717e57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:22.414430 21673 consensus_queue.cc:1035] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0c83b31373ec423c8f4524befc90bfc4" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:22.414683 21677 consensus_queue.cc:1035] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b8612fb089f409d9faffefe7c9cdec4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:22.417582 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.416860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:34053)
0928 14:46:22.417420 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.417491 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462}
I20260928 14:46:22.418100 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.417197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:43763)
0928 14:46:22.417966 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.418024 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":692}
I20260928 14:46:22.421284 21648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.420794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.66:43193)
0928 14:46:22.421102 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.421167 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230}
I20260928 14:46:22.421491 21646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.421040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.67:52223)
0928 14:46:22.421340 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.421409 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219}
I20260928 14:46:22.424192 21647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.423825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44095 (local address 127.13.223.65:58787)
0928 14:46:22.424036 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.424115 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44095: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20260928 14:46:22.430177 21105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:22.430501 21105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:22.435775 21105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a801913b65ef454fa708392b7df506ea: Sending DeleteTablet for 3 replicas of tablet d012cb7b506e4caaa14e05965abfefe5
I20260928 14:46:22.436626 21498 tablet_service.cc:1514] Processing DeleteTablet for tablet d012cb7b506e4caaa14e05965abfefe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:22 UTC) from {username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:36728
I20260928 14:46:22.436638 21224 tablet_service.cc:1514] Processing DeleteTablet for tablet d012cb7b506e4caaa14e05965abfefe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:22 UTC) from {username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:51484
I20260928 14:46:22.437121 21691 tablet_replica.cc:330] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: stopping tablet replica
I20260928 14:46:22.437288 21691 raft_consensus.cc:2229] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:22.437448 21691 raft_consensus.cc:2258] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:22.437461 21361 tablet_service.cc:1514] Processing DeleteTablet for tablet d012cb7b506e4caaa14e05965abfefe5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:22 UTC) from {username='oryx', principal='oryx/127.13.223.124@KRBTEST.COM'} at 127.0.0.1:57970
I20260928 14:46:22.437711 21690 tablet_replica.cc:330] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: stopping tablet replica
I20260928 14:46:22.437835 21690 raft_consensus.cc:2229] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:22.437992 21690 raft_consensus.cc:2258] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:22.438416 21692 tablet_replica.cc:330] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: stopping tablet replica
I20260928 14:46:22.438540 21692 raft_consensus.cc:2229] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:22.438735 21691 ts_tablet_manager.cc:1905] T d012cb7b506e4caaa14e05965abfefe5 P 0c83b31373ec423c8f4524befc90bfc4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:22.438874 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21072
I20260928 14:46:22.441239 21690 ts_tablet_manager.cc:1905] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:22.443603 21690 ts_tablet_manager.cc:1918] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:22.443709 21690 log.cc:1196] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/d012cb7b506e4caaa14e05965abfefe5
I20260928 14:46:22.443986 21690 ts_tablet_manager.cc:1939] T d012cb7b506e4caaa14e05965abfefe5 P 4b8612fb089f409d9faffefe7c9cdec4: Deleting consensus metadata
I20260928 14:46:22.447026 21098 catalog_manager.cc:4841] TS 4b8612fb089f409d9faffefe7c9cdec4 (127.13.223.67:45707): tablet d012cb7b506e4caaa14e05965abfefe5 (table test-table [id=ab9b3458ce074b15833228a80401df3e]) successfully deleted
I20260928 14:46:22.450623 21692 raft_consensus.cc:2258] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:22.451637 21692 ts_tablet_manager.cc:1905] T d012cb7b506e4caaa14e05965abfefe5 P 504a5636894b4f94bbae983fd2717e57: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:46:22.451855 21665 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:22.451092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:41187 (local address 127.0.0.1:51490)
0928 14:46:22.451284 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.451299 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:22.451386 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:22.451740 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:41187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":110}
W20260928 14:46:22.452077 21097 catalog_manager.cc:4584] TS 0c83b31373ec423c8f4524befc90bfc4 (127.13.223.65:41187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d012cb7b506e4caaa14e05965abfefe5: Network error: Client connection negotiation failed: client connection to 127.13.223.65:41187: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:46:22.452229 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21298
I20260928 14:46:22.459314 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21435
I20260928 14:46:22.459946 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:22.459566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41435 (local address 127.0.0.1:57986)
0928 14:46:22.459772 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:22.459835 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:41435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260928 14:46:22.467748 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20790
I20260928 14:46:22.474345 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 20892
2026-09-28T14:46:22Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7553 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:22 dist-test-slave-gkhx krb5kdc[21695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:22 dist-test-slave-gkhx krb5kdc[21695](info): set up 2 sockets
Sep 28 14:46:22 dist-test-slave-gkhx krb5kdc[21695](info): commencing operation
krb5kdc: starting...
W20260928 14:46:23.626024 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.125s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:23 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab.
Entry for principal oryx/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab.
I20260928 14:46:23.767509 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35795
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab
--principal=oryx/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:35795
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:23.889047 21708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:23.889376 21708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:23.889420 21708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:23.894271 21708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:23.894374 21708 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 14:46:23.894402 21708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:23.894429 21708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:23.894451 21708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:23.894483 21708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:23.900933 21708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:35795
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.126.keytab
--principal=oryx/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:23.902119 21708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:23.903249 21708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:23.912846 21716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:23.912827 21713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:23.912900 21708 server_base.cc:1034] running on GCE node
W20260928 14:46:23.912952 21714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:23.913771 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:23.915169 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:23.916353 21708 hybrid_clock.cc:648] HybridClock initialized: now 1790606783916351 us; error 50 us; skew 500 ppm
Sep 28 14:46:23 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606783, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:23.919546 21708 init.cc:377] Logged in from keytab as oryx/127.13.223.126@KRBTEST.COM (short username oryx)
I20260928 14:46:23.920662 21708 webserver.cc:460] Webserver started at http://127.13.223.126:39681/ using document root <none> and password file <none>
I20260928 14:46:23.921296 21708 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:23.921408 21708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:23.921671 21708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:23.924036 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "869a44c876e642e686e6a83b7cc32558"
format_stamp: "Formatted at 2026-09-28 14:46:23 on dist-test-slave-gkhx"
server_key: "4a56c675965348fc146e69904dd5ee3d"
server_key_iv: "9103cb16c73785882878b409a893bb2f"
server_key_version: "encryptionkey@0"
I20260928 14:46:23.924783 21708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "869a44c876e642e686e6a83b7cc32558"
format_stamp: "Formatted at 2026-09-28 14:46:23 on dist-test-slave-gkhx"
server_key: "4a56c675965348fc146e69904dd5ee3d"
server_key_iv: "9103cb16c73785882878b409a893bb2f"
server_key_version: "encryptionkey@0"
I20260928 14:46:23.929206 21708 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:23.932243 21723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:23.933893 21708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:46:23.934146 21708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "869a44c876e642e686e6a83b7cc32558"
format_stamp: "Formatted at 2026-09-28 14:46:23 on dist-test-slave-gkhx"
server_key: "4a56c675965348fc146e69904dd5ee3d"
server_key_iv: "9103cb16c73785882878b409a893bb2f"
server_key_version: "encryptionkey@0"
I20260928 14:46:23.934324 21708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:23.970057 21708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:23.974061 21708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:23.974341 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:23.983209 21708 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:35795
I20260928 14:46:23.983284 21785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:35795 every 8 connection(s)
I20260928 14:46:23.984308 21708 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:23.986132 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21708
I20260928 14:46:23.986281 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:23.986675 14205 external_mini_cluster.cc:1428] Setting key 607cec5fbc7962d63e4443ba67ffc417
I20260928 14:46:23.987643 21786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:23.995065 21786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Bootstrap starting.
Sep 28 14:46:23 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:23.997900 21786 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:23.999471 21786 log.cc:824] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:24.002238 21786 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: No bootstrap required, opened a new log
I20260928 14:46:24.004084 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:23.988161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51522 (local address 127.13.223.126:35795)
0928 14:46:23.988624 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:23.988645 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:23.988690 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:46:23.989579 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:23.989603 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:23.990062 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:23.990489 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:23.991345 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:23.992874 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:23.993925 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:23.994674 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:23.994841 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:23.997960 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:23.997969 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:23.997989 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:23.998032 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.000281 (+ 2249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.000831 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.000838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.000845 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.001010 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.001373 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.001376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.001378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.001972 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.002189 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.002730 (+ 541us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.003171 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":130,"threads_started":1}
I20260928 14:46:24.005719 21786 raft_consensus.cc:348] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } }
I20260928 14:46:24.005999 21786 raft_consensus.cc:374] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:24.006083 21786 raft_consensus.cc:729] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869a44c876e642e686e6a83b7cc32558, State: Initialized, Role: FOLLOWER
I20260928 14:46:24.006597 21786 consensus_queue.cc:260] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } }
I20260928 14:46:24.006773 21786 raft_consensus.cc:388] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:24.006837 21786 raft_consensus.cc:482] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:24.006948 21786 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:24.008035 21786 raft_consensus.cc:504] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } }
I20260928 14:46:24.008425 21786 leader_election.cc:302] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 869a44c876e642e686e6a83b7cc32558; no voters:
I20260928 14:46:24.008802 21786 leader_election.cc:288] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:24.009063 21791 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:24.009315 21791 raft_consensus.cc:686] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [term 1 LEADER]: Becoming Leader. State: Replica: 869a44c876e642e686e6a83b7cc32558, State: Running, Role: LEADER
I20260928 14:46:24.009678 21791 consensus_queue.cc:237] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } }
I20260928 14:46:24.010285 21786 sys_catalog.cc:564] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:24.011693 21793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 869a44c876e642e686e6a83b7cc32558. Latest consensus state: current_term: 1 leader_uuid: "869a44c876e642e686e6a83b7cc32558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } } }
I20260928 14:46:24.011790 21792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "869a44c876e642e686e6a83b7cc32558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "869a44c876e642e686e6a83b7cc32558" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35795 } } }
I20260928 14:46:24.011914 21792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:24.011859 21793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:24.012384 21800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:24.015908 21800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:24.023478 21800 catalog_manager.cc:1347] Generated new cluster ID: 754739a537c243ef950a6b87a7594b07
I20260928 14:46:24.023586 21800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:24.050228 21800 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:24.051631 21800 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:24.057772 21800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Generated new TSK 0
I20260928 14:46:24.058784 21800 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
I20260928 14:46:24.094810 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35795
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:24.219441 21812 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:24.219753 21812 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:24.219802 21812 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:24.224295 21812 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:24.224381 21812 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:24.224427 21812 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:46:24.224463 21812 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:24.230278 21812 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21812
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:24.231587 21812 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:24.232649 21812 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:24.243609 21820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:24.243609 21818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:24.243845 21812 server_base.cc:1034] running on GCE node
W20260928 14:46:24.243659 21817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:24.244500 21812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:24.245282 21812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:24.246614 21812 hybrid_clock.cc:648] HybridClock initialized: now 1790606784246582 us; error 42 us; skew 500 ppm
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:24.250005 21812 init.cc:377] Logged in from keytab as oryx/127.13.223.65@KRBTEST.COM (short username oryx)
I20260928 14:46:24.251219 21812 webserver.cc:460] Webserver started at http://127.13.223.65:33293/ using document root <none> and password file <none>
I20260928 14:46:24.251965 21812 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:24.252045 21812 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:24.252326 21812 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:24.254863 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "59656799d778444c877497c40f599ee9"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "244a3baed747f6008868f86fef33d981"
server_key_iv: "29061788b5aedf54d21da719b85f94d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.255548 21812 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "59656799d778444c877497c40f599ee9"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "244a3baed747f6008868f86fef33d981"
server_key_iv: "29061788b5aedf54d21da719b85f94d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.260186 21812 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:24.263185 21827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.264606 21812 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:24.264762 21812 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "59656799d778444c877497c40f599ee9"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "244a3baed747f6008868f86fef33d981"
server_key_iv: "29061788b5aedf54d21da719b85f94d9"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.264897 21812 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:24.294755 21812 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:24.298754 21812 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:24.299005 21812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:24.300630 21812 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:24.300688 21812 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.300731 21812 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:24.300798 21812 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.312759 21812 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:44515
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:24.312870 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:44515 every 8 connection(s)
I20260928 14:46:24.313881 21812 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:24.317854 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.301941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:57905 (local address 127.13.223.126:35795)
0928 14:46:24.302120 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.302127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.303222 (+ 1095us) server_negotiation.cc:407] Connection header received
0928 14:46:24.304554 (+ 1332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.304558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.304633 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.304734 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.306771 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.307504 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.309601 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.310004 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.310103 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.313677 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.313681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.313683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.313713 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.315707 (+ 1994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.316363 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.316366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.316367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.316469 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.316795 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.316798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.316800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.317014 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.317093 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.317598 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.317720 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:46:24.318809 21836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.302436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.65:57905)
0928 14:46:24.303034 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.303095 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.304291 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.304993 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.305002 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.305612 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.306505 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.306533 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.307666 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.307672 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.309391 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.309404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.310198 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.310202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.310492 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.311454 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.311474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.313435 (+ 1961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.315847 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.315852 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.315864 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.316199 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.316556 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.316558 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.316559 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.316692 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.317106 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.317111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.317357 (+ 246us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.317520 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.317791 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":167,"threads_started":1}
I20260928 14:46:24.320366 21943 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35795
I20260928 14:46:24.320549 21943 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:24.321148 21943 heartbeater.cc:511] Master 127.13.223.126:35795 requested a full tablet report, sending...
I20260928 14:46:24.322965 21738 ts_manager.cc:194] Registered new tserver with Master: 59656799d778444c877497c40f599ee9 (127.13.223.65:44515)
I20260928 14:46:24.323493 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21812
I20260928 14:46:24.323649 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:24.323989 14205 external_mini_cluster.cc:1428] Setting key 0e601184fd6ddc2aa242d245c519f3ab
I20260928 14:46:24.324357 21738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:57905
I20260928 14:46:24.334224 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.322655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:44771 (local address 127.13.223.126:35795)
0928 14:46:24.322779 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.322782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.325274 (+ 2492us) server_negotiation.cc:407] Connection header received
0928 14:46:24.325529 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.325535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.325586 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.325663 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.326274 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.327155 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.328119 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.328555 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.328693 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.330197 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.330200 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.330202 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.330228 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.332061 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.332752 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.332755 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.332756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.332903 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.333276 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.333279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.333280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.333494 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.333577 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.333972 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.334094 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:46:24.334394 21949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.322528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.65:44771)
0928 14:46:24.325171 (+ 2643us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.325184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.325356 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.325693 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.325696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.325970 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.326100 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.326109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.327330 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.327335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.327981 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.327990 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.328722 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.328725 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.328836 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.329449 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.329471 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.330029 (+ 558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.332208 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.332216 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.332222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.332576 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.333021 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.333028 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.333034 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.333159 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.333610 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.333616 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.333752 (+ 136us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.333884 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.334096 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2547,"thread_start_us":139,"threads_started":1}
WARNING: no policy specified for oryx/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
I20260928 14:46:24.358873 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35795
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:24.483474 21952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:24.483862 21952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:24.483923 21952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:24.488984 21952 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:24.489076 21952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:24.489123 21952 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:46:24.489163 21952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:24.495805 21952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21952
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:24.497123 21952 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:24.498368 21952 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:24.509778 21957 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:24.509788 21960 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:24.509799 21958 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:24.510183 21952 server_base.cc:1034] running on GCE node
I20260928 14:46:24.510850 21952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:24.511708 21952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:24.512900 21952 hybrid_clock.cc:648] HybridClock initialized: now 1790606784512880 us; error 35 us; skew 500 ppm
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:24.516325 21952 init.cc:377] Logged in from keytab as oryx/127.13.223.66@KRBTEST.COM (short username oryx)
I20260928 14:46:24.517359 21952 webserver.cc:460] Webserver started at http://127.13.223.66:43391/ using document root <none> and password file <none>
I20260928 14:46:24.517993 21952 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:24.518093 21952 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:24.518337 21952 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:24.520756 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "c86abce5dca142a8842a3a5358d0a558"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "58571215a7d13b87132f268a1682c2e3"
server_key_iv: "44910f8c206b714ed82337ebae235658"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.521364 21952 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "c86abce5dca142a8842a3a5358d0a558"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "58571215a7d13b87132f268a1682c2e3"
server_key_iv: "44910f8c206b714ed82337ebae235658"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.525636 21952 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:46:24.528898 21967 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.530411 21952 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:46:24.530611 21952 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "c86abce5dca142a8842a3a5358d0a558"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "58571215a7d13b87132f268a1682c2e3"
server_key_iv: "44910f8c206b714ed82337ebae235658"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.530771 21952 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:24.554723 21952 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:24.558537 21952 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:24.558843 21952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:24.560647 21952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:24.560750 21952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.560835 21952 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:24.560897 21952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:24.573459 21952 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:38703
I20260928 14:46:24.573683 22082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:38703 every 8 connection(s)
I20260928 14:46:24.574944 21952 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:24.577265 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 21952
I20260928 14:46:24.577432 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:24.577495 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.561771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:59999 (local address 127.13.223.126:35795)
0928 14:46:24.561955 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.561962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.562925 (+ 963us) server_negotiation.cc:407] Connection header received
0928 14:46:24.564026 (+ 1101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.564031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.564102 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.564172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.566206 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.567314 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.568809 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.569196 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.569330 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.572652 (+ 3322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.572658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.572662 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.572705 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.574757 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.575651 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.575655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.575657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.575774 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.576181 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.576185 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.576186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.576422 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.576519 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.577169 (+ 650us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.577324 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:24.577788 14205 external_mini_cluster.cc:1428] Setting key 727d383f8dfb11ad39050ca03ca8e8c9
I20260928 14:46:24.578598 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.562191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.66:59999)
0928 14:46:24.562732 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.562787 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.563715 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.564674 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.564685 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.565248 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.565992 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.566017 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.567472 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.567476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.568662 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.568674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.569390 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.569394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.569592 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.570294 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.570319 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.572396 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.575043 (+ 2647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.575049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.575096 (+ 47us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.575503 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.575891 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.575896 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.575901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.576062 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.576567 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.576572 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.576844 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.577119 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.577497 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":156,"threads_started":1}
I20260928 14:46:24.580055 22083 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35795
I20260928 14:46:24.580312 22083 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:24.581055 22083 heartbeater.cc:511] Master 127.13.223.126:35795 requested a full tablet report, sending...
I20260928 14:46:24.582477 21738 ts_manager.cc:194] Registered new tserver with Master: c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703)
I20260928 14:46:24.583451 21738 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:59999
WARNING: no policy specified for oryx/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.67@KRBTEST.COM" created.
I20260928 14:46:24.593500 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.582330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.66:40137)
0928 14:46:24.582741 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.582754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.582859 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.584224 (+ 1365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.584229 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.584510 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.584696 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.584708 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.585909 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.585915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.587013 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.587028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.587573 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.587576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.587690 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.588267 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.588288 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.588770 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.591391 (+ 2621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.591397 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.591399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.591643 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.592131 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.592134 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.592136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.592263 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.592849 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.592852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.592971 (+ 119us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.593095 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.593283 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":1,"thread_start_us":152,"threads_started":1}
I20260928 14:46:24.593489 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.582345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:40137 (local address 127.13.223.126:35795)
0928 14:46:24.583836 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.583843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.583861 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:24.583912 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.583917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.583969 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.584069 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.584861 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.585722 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.587166 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.587424 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.587526 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.589179 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.589184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.589186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.589216 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.591160 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.591762 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.591765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.591767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.591982 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.592401 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.592420 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.592425 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.592723 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.592855 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.593152 (+ 297us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.593280 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1404}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
I20260928 14:46:24.615387 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35795
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:36665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:24.741714 22092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:24.742058 22092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:24.742108 22092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:24.747269 22092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:24.747365 22092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:24.747417 22092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 14:46:24.747457 22092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:24.753746 22092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:36665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.13.223.126:35795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:24.755041 22092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:24.756138 22092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:24.767388 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:24.767390 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:24.767390 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:24.767941 22092 server_base.cc:1034] running on GCE node
I20260928 14:46:24.768471 22092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:24.769346 22092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:24.770821 22092 hybrid_clock.cc:648] HybridClock initialized: now 1790606784770811 us; error 118 us; skew 500 ppm
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:24.774281 22092 init.cc:377] Logged in from keytab as oryx/127.13.223.67@KRBTEST.COM (short username oryx)
I20260928 14:46:24.775470 22092 webserver.cc:460] Webserver started at http://127.13.223.67:37421/ using document root <none> and password file <none>
I20260928 14:46:24.776077 22092 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:24.776144 22092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:24.776335 22092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:24.778680 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "30ba036f00c94830a4f5764eafee36ba"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "e95dbb4570f21ad0fd2e6cec6c5c4e3a"
server_key_iv: "9f28f021bd324cd091d4bf5168b04300"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.779289 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "30ba036f00c94830a4f5764eafee36ba"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "e95dbb4570f21ad0fd2e6cec6c5c4e3a"
server_key_iv: "9f28f021bd324cd091d4bf5168b04300"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.783922 22092 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:46:24.786986 22107 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.788609 22092 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:24.788789 22092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "30ba036f00c94830a4f5764eafee36ba"
format_stamp: "Formatted at 2026-09-28 14:46:24 on dist-test-slave-gkhx"
server_key: "e95dbb4570f21ad0fd2e6cec6c5c4e3a"
server_key_iv: "9f28f021bd324cd091d4bf5168b04300"
server_key_version: "encryptionkey@0"
I20260928 14:46:24.788997 22092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:24.800230 22092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:24.804165 22092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:24.804401 22092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:24.806046 22092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:24.806136 22092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:24.806213 22092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:24.806245 22092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:24.817760 22092 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:39649
I20260928 14:46:24.817984 22222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:39649 every 8 connection(s)
I20260928 14:46:24.818917 22092 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:24.823377 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.807157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44409 (local address 127.13.223.126:35795)
0928 14:46:24.807328 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.807334 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.808297 (+ 963us) server_negotiation.cc:407] Connection header received
0928 14:46:24.809780 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.809787 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.809873 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.809999 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.811949 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.812570 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.814262 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.814716 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.814855 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.818642 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.818652 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.818654 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.818688 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.820701 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.821326 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.821329 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.821330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.821423 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.821880 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.821884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.821885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.822123 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.822308 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.823094 (+ 786us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.823232 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:46:24.823709 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22092
I20260928 14:46:24.823820 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:24.824150 14205 external_mini_cluster.cc:1428] Setting key c377916f5ad830fad70446c646766410
I20260928 14:46:24.824438 22116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.807571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.67:44409)
0928 14:46:24.808090 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.808149 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.809417 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.810266 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.810276 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.811000 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.811732 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.811757 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.812760 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.812767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.814091 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.814105 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.814912 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.814917 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.815145 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.815946 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.815978 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.818336 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.820837 (+ 2501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.820843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.820858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.821204 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.821521 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.821523 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.821525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.821738 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.822328 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.822335 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.822741 (+ 406us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.822925 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.823249 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":152,"threads_started":1}
I20260928 14:46:24.825879 22223 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35795
I20260928 14:46:24.826092 22223 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:24.826773 22223 heartbeater.cc:511] Master 127.13.223.126:35795 requested a full tablet report, sending...
I20260928 14:46:24.827924 21735 ts_manager.cc:194] Registered new tserver with Master: 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:24.828784 21735 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:44409
I20260928 14:46:24.836207 22229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.828227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.13.223.67:52099)
0928 14:46:24.828535 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.828555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.828639 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.828904 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.828908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.829167 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:24.829296 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.829302 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.830086 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.830090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.830963 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.830975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.831423 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.831426 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.831507 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.832086 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:24.832098 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:24.832455 (+ 357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:24.834229 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.834233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.834235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.834587 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.835028 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:24.835031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:24.835032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:24.835129 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:24.835650 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:24.835655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:24.835764 (+ 109us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.835864 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.836019 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":106,"threads_started":1}
I20260928 14:46:24.836222 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.828342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:52099 (local address 127.13.223.126:35795)
0928 14:46:24.828478 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.828483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.828620 (+ 137us) server_negotiation.cc:407] Connection header received
0928 14:46:24.828725 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.828728 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.828775 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.828859 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.829417 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.829950 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.831093 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.831317 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.831403 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.832588 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.832591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.832593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.832616 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.834083 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.834736 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.834746 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.834748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.834859 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.835235 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.835238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.835240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.835511 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.835596 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.835965 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.836093 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:46:24.839308 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:24.867982 21789 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:24.860353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.13.223.126:35795)
0928 14:46:24.860494 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.860497 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.860511 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:46:24.860554 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.860556 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.860596 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.860671 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.861761 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.862410 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.863590 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.863864 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.863971 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.867787 (+ 3816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51532: BlockingRecv error: recv got EOF from 127.0.0.1:51532 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:46:24 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606784, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:24.879575 21789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.869734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.13.223.126:35795)
0928 14:46:24.869890 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.869895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.870134 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:46:24.870273 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.870277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.870333 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.870424 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:24.871182 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.871752 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.872628 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.872872 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.872996 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.875730 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:24.875734 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:24.875736 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:24.875775 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:24.877321 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.877911 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.877914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.877915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.878010 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:24.878329 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:24.878333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:24.878335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:24.878625 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:24.878779 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.879209 (+ 430us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.879373 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:46:24.883591 21735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:24.887249 21735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:24.906100 22245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.898577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:39649 (local address 127.0.0.1:49992)
0928 14:46:24.900073 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.900137 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.900279 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.901121 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.901130 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.901160 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:24.901313 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.901324 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.902850 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.902853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.904207 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.904219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905402 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.905541 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905583 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.905716 (+ 133us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.905895 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1319,"mutex_wait_us":116,"thread_start_us":106,"threads_started":1}
I20260928 14:46:24.906495 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.898942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39208 (local address 127.13.223.66:38703)
0928 14:46:24.899818 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.899833 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.900760 (+ 927us) server_negotiation.cc:407] Connection header received
0928 14:46:24.900849 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.900857 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.901062 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.901210 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:24.901626 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.902892 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.904712 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905351 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905461 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905623 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.906210 (+ 587us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.906358 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":756,"thread_start_us":96,"threads_started":1}
I20260928 14:46:24.906512 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.899754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51956 (local address 127.13.223.65:44515)
0928 14:46:24.900396 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.900402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.900915 (+ 513us) server_negotiation.cc:407] Connection header received
0928 14:46:24.901241 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.901248 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.901597 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.901754 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:24.901986 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.903109 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.904673 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905464 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905587 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905738 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.906231 (+ 493us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.906345 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":89,"threads_started":1}
I20260928 14:46:24.906100 22249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.898766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:38703 (local address 127.0.0.1:39208)
0928 14:46:24.900554 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.900568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.900662 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.901346 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.901351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.901372 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:24.901499 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.901509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.903032 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.903035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.904567 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.904577 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905685 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.905789 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905810 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.905875 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.905929 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1725,"mutex_wait_us":52,"thread_start_us":110,"threads_started":1}
I20260928 14:46:24.907290 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.899351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.13.223.67:39649)
0928 14:46:24.900158 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:24.900172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:24.900203 (+ 31us) server_negotiation.cc:407] Connection header received
0928 14:46:24.900526 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:24.900532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:24.900775 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:24.900929 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:24.901543 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.902706 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.904355 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905211 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905342 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905538 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:24.905773 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 14:46:24.907105 (+ 1332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":107,"threads_started":1}
I20260928 14:46:24.907426 22246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:24.899596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44515 (local address 127.0.0.1:51956)
0928 14:46:24.900737 (+ 1141us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:24.900752 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:24.901008 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:24.901741 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:24.901745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:24.901763 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:24.901855 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.901861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.903232 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.903236 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.904523 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:24.904536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:24.905809 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:24.905814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:24.905934 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:24.905959 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:46:24.906030 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 14:46:24.906113 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":63,"thread_start_us":86,"threads_started":1}
I20260928 14:46:24.910296 22157 tablet_service.cc:1467] Processing CreateTablet for tablet d00c51dfa1704930baea7a0156151f0f (DEFAULT_TABLE table=test-table [id=0d79b38394874dbba499bc19676d9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:24.910337 21877 tablet_service.cc:1467] Processing CreateTablet for tablet d00c51dfa1704930baea7a0156151f0f (DEFAULT_TABLE table=test-table [id=0d79b38394874dbba499bc19676d9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:24.911636 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00c51dfa1704930baea7a0156151f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:24.911651 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00c51dfa1704930baea7a0156151f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:24.910373 22017 tablet_service.cc:1467] Processing CreateTablet for tablet d00c51dfa1704930baea7a0156151f0f (DEFAULT_TABLE table=test-table [id=0d79b38394874dbba499bc19676d9dde]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:24.913327 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d00c51dfa1704930baea7a0156151f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:24.919992 22251 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Bootstrap starting.
I20260928 14:46:24.921730 22252 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Bootstrap starting.
I20260928 14:46:24.923035 22251 tablet_bootstrap.cc:654] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:24.923914 22251 log.cc:824] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:24.924335 22252 tablet_bootstrap.cc:654] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:24.925338 22252 log.cc:824] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:24.925377 22254 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Bootstrap starting.
I20260928 14:46:24.926482 22251 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: No bootstrap required, opened a new log
I20260928 14:46:24.926788 22251 ts_tablet_manager.cc:1397] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:24.927979 22252 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: No bootstrap required, opened a new log
I20260928 14:46:24.928105 22254 tablet_bootstrap.cc:654] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:24.928246 22252 ts_tablet_manager.cc:1397] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:46:24.929097 22254 log.cc:824] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:24.931854 22251 raft_consensus.cc:348] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.932265 22251 raft_consensus.cc:374] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:24.932324 22251 raft_consensus.cc:729] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba036f00c94830a4f5764eafee36ba, State: Initialized, Role: FOLLOWER
I20260928 14:46:24.932816 22251 consensus_queue.cc:260] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.932978 22254 tablet_bootstrap.cc:492] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: No bootstrap required, opened a new log
I20260928 14:46:24.933226 22254 ts_tablet_manager.cc:1397] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260928 14:46:24.933693 22223 heartbeater.cc:503] Master 127.13.223.126:35795 was elected leader, sending a full tablet report...
I20260928 14:46:24.933981 22251 ts_tablet_manager.cc:1428] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:24.933354 22252 raft_consensus.cc:348] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.934327 22252 raft_consensus.cc:374] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:24.934403 22252 raft_consensus.cc:729] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59656799d778444c877497c40f599ee9, State: Initialized, Role: FOLLOWER
I20260928 14:46:24.935370 22252 consensus_queue.cc:260] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.936595 21943 heartbeater.cc:503] Master 127.13.223.126:35795 was elected leader, sending a full tablet report...
I20260928 14:46:24.936991 22252 ts_tablet_manager.cc:1428] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260928 14:46:24.938094 22254 raft_consensus.cc:348] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.938467 22254 raft_consensus.cc:374] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:24.938581 22254 raft_consensus.cc:729] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Initialized, Role: FOLLOWER
I20260928 14:46:24.939144 22254 consensus_queue.cc:260] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:24.940407 22083 heartbeater.cc:503] Master 127.13.223.126:35795 was elected leader, sending a full tablet report...
I20260928 14:46:24.940476 22254 ts_tablet_manager.cc:1428] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
W20260928 14:46:25.065538 21944 tablet.cc:2369] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:46:25.070374 22224 tablet.cc:2369] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:46:25.076566 22084 tablet.cc:2369] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:46:25.080850 22259 raft_consensus.cc:482] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:25.081081 22259 raft_consensus.cc:504] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:25.082322 22259 leader_election.cc:288] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649), 59656799d778444c877497c40f599ee9 (127.13.223.65:44515)
I20260928 14:46:25.088522 22250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.082756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44799 (local address 127.13.223.65:44515)
0928 14:46:25.082946 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.082952 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.083353 (+ 401us) server_negotiation.cc:407] Connection header received
0928 14:46:25.083432 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.083435 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.083494 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.083588 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:25.084081 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.084942 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.086898 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.087644 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.087799 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.087859 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.088308 (+ 449us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.088407 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260928 14:46:25.088513 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.082801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:54925 (local address 127.13.223.67:39649)
0928 14:46:25.082928 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.082933 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.083365 (+ 432us) server_negotiation.cc:407] Connection header received
0928 14:46:25.083432 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.083436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.083494 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.083588 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:25.084055 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.084894 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.086898 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.087612 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.087799 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.087859 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.088254 (+ 395us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.088359 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 14:46:25.088511 22261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.082680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:39649 (local address 127.13.223.66:54925)
0928 14:46:25.083217 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.083228 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.083337 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.083622 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.083626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.083669 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:25.083879 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.083888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.085082 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.085086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.086665 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.086679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.087817 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.087822 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.088030 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.088096 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.088176 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.088327 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":173,"spinlock_wait_cycles":32256,"thread_start_us":110,"threads_started":1}
I20260928 14:46:25.088511 22260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.082671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44515 (local address 127.13.223.66:44799)
0928 14:46:25.083119 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.083138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.083286 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.083622 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.083626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.083671 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:25.083952 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.083961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.085082 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.085086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.086665 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.086679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.087818 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.087822 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.088034 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.088077 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.088176 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.088301 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":301,"spinlock_wait_cycles":16384,"thread_start_us":133,"threads_started":1}
I20260928 14:46:25.089563 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00c51dfa1704930baea7a0156151f0f" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba" is_pre_election: true
I20260928 14:46:25.089566 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00c51dfa1704930baea7a0156151f0f" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9" is_pre_election: true
I20260928 14:46:25.089941 22172 raft_consensus.cc:2454] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:25.089941 21897 raft_consensus.cc:2454] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:25.090687 21970 leader_election.cc:302] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, 59656799d778444c877497c40f599ee9, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:25.091074 22259 raft_consensus.cc:2781] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:25.091200 22259 raft_consensus.cc:482] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:25.091290 22259 raft_consensus.cc:3037] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.093417 22259 raft_consensus.cc:504] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:25.093941 22259 leader_election.cc:288] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Requested vote from peers 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649), 59656799d778444c877497c40f599ee9 (127.13.223.65:44515)
I20260928 14:46:25.094379 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00c51dfa1704930baea7a0156151f0f" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba"
I20260928 14:46:25.094480 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d00c51dfa1704930baea7a0156151f0f" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9"
I20260928 14:46:25.094585 22172 raft_consensus.cc:3037] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.094638 21897 raft_consensus.cc:3037] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.096151 21897 raft_consensus.cc:2454] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:25.096578 22172 raft_consensus.cc:2454] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:25.096623 21970 leader_election.cc:302] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59656799d778444c877497c40f599ee9, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:25.096830 22259 raft_consensus.cc:2781] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:25.097134 22259 raft_consensus.cc:686] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 1 LEADER]: Becoming Leader. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Running, Role: LEADER
I20260928 14:46:25.097544 22259 consensus_queue.cc:237] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } }
I20260928 14:46:25.101697 21738 catalog_manager.cc:5495] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86abce5dca142a8842a3a5358d0a558 (127.13.223.66). New cstate: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:25.149317 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.144513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39214 (local address 127.13.223.66:38703)
0928 14:46:25.144661 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.144665 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.144682 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:25.144770 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.144773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.144830 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.144951 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:25.145281 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.145944 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.147274 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.147524 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.147622 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.148028 (+ 406us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.148580 (+ 552us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.148729 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.148994 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.149108 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:46:25.156178 21897 raft_consensus.cc:1261] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.156178 22172 raft_consensus.cc:1261] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.156981 22259 consensus_queue.cc:1035] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.157223 22262 consensus_queue.cc:1035] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.168139 22266 mvcc.cc:205] Tried to move back new op lower bound from 7334325391991615488 to 7334325391762890752. Current Snapshot: MvccSnapshot[applied={T|T < 7334325391991615488}]
I20260928 14:46:25.175122 21740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:25.175318 21740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:25.178735 21740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Sending DeleteTablet for 3 replicas of tablet d00c51dfa1704930baea7a0156151f0f
I20260928 14:46:25.179793 22157 tablet_service.cc:1514] Processing DeleteTablet for tablet d00c51dfa1704930baea7a0156151f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49992
I20260928 14:46:25.179898 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet d00c51dfa1704930baea7a0156151f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39208
I20260928 14:46:25.180074 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet d00c51dfa1704930baea7a0156151f0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51956
I20260928 14:46:25.180599 22275 tablet_replica.cc:330] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: stopping tablet replica
I20260928 14:46:25.180652 22276 tablet_replica.cc:330] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: stopping tablet replica
I20260928 14:46:25.181079 22275 raft_consensus.cc:2229] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:25.181087 22276 raft_consensus.cc:2229] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:25.181217 21740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:46:25.181502 22275 raft_consensus.cc:2258] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:46:25.181753 21740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:25.181965 22276 raft_consensus.cc:2258] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:25.182427 22277 tablet_replica.cc:330] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: stopping tablet replica
I20260928 14:46:25.182868 22277 raft_consensus.cc:2229] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:25.183260 22277 raft_consensus.cc:2258] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:25.183840 22275 ts_tablet_manager.cc:1905] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.184659 22276 ts_tablet_manager.cc:1905] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.186818 22277 ts_tablet_manager.cc:1905] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.188413 21877 tablet_service.cc:1467] Processing CreateTablet for tablet e6835167d4e24ad595cede1810f2156f (DEFAULT_TABLE table=test-table [id=642dc9d034c144fca4bfadcbe5f14167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.188731 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835167d4e24ad595cede1810f2156f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.190102 22277 ts_tablet_manager.cc:1918] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:25.190301 22277 log.cc:1196] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/d00c51dfa1704930baea7a0156151f0f
I20260928 14:46:25.190639 22276 ts_tablet_manager.cc:1918] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:25.190750 22276 log.cc:1196] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/d00c51dfa1704930baea7a0156151f0f
I20260928 14:46:25.190827 22017 tablet_service.cc:1467] Processing CreateTablet for tablet e6835167d4e24ad595cede1810f2156f (DEFAULT_TABLE table=test-table [id=642dc9d034c144fca4bfadcbe5f14167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.190881 22277 ts_tablet_manager.cc:1939] T d00c51dfa1704930baea7a0156151f0f P 59656799d778444c877497c40f599ee9: Deleting consensus metadata
I20260928 14:46:25.191130 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835167d4e24ad595cede1810f2156f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.192018 22276 ts_tablet_manager.cc:1939] T d00c51dfa1704930baea7a0156151f0f P c86abce5dca142a8842a3a5358d0a558: Deleting consensus metadata
I20260928 14:46:25.193315 21726 catalog_manager.cc:4841] TS c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703): tablet d00c51dfa1704930baea7a0156151f0f (table test-table [id=0d79b38394874dbba499bc19676d9dde]) successfully deleted
I20260928 14:46:25.193542 21726 catalog_manager.cc:4841] TS 59656799d778444c877497c40f599ee9 (127.13.223.65:44515): tablet d00c51dfa1704930baea7a0156151f0f (table test-table [id=0d79b38394874dbba499bc19676d9dde]) successfully deleted
I20260928 14:46:25.193751 22254 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Bootstrap starting.
I20260928 14:46:25.193682 22157 tablet_service.cc:1467] Processing CreateTablet for tablet e6835167d4e24ad595cede1810f2156f (DEFAULT_TABLE table=test-table [id=642dc9d034c144fca4bfadcbe5f14167]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.194007 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6835167d4e24ad595cede1810f2156f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.195139 22254 tablet_bootstrap.cc:654] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.196631 22254 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: No bootstrap required, opened a new log
I20260928 14:46:25.196730 22254 ts_tablet_manager.cc:1397] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:25.196861 22251 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Bootstrap starting.
I20260928 14:46:25.197250 22254 raft_consensus.cc:348] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.197376 22254 raft_consensus.cc:374] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.197409 22254 raft_consensus.cc:729] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.197770 22254 consensus_queue.cc:260] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.197923 22252 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Bootstrap starting.
I20260928 14:46:25.197958 22251 tablet_bootstrap.cc:654] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.199052 22252 tablet_bootstrap.cc:654] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.199772 22275 ts_tablet_manager.cc:1918] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:25.199891 22275 log.cc:1196] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/d00c51dfa1704930baea7a0156151f0f
I20260928 14:46:25.200192 22254 ts_tablet_manager.cc:1428] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:25.200250 22275 ts_tablet_manager.cc:1939] T d00c51dfa1704930baea7a0156151f0f P 30ba036f00c94830a4f5764eafee36ba: Deleting consensus metadata
I20260928 14:46:25.201798 22251 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: No bootstrap required, opened a new log
I20260928 14:46:25.201897 22251 ts_tablet_manager.cc:1397] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 14:46:25.201978 21727 catalog_manager.cc:4841] TS 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649): tablet d00c51dfa1704930baea7a0156151f0f (table test-table [id=0d79b38394874dbba499bc19676d9dde]) successfully deleted
I20260928 14:46:25.202425 22251 raft_consensus.cc:348] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.202595 22251 raft_consensus.cc:374] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.202646 22251 raft_consensus.cc:729] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba036f00c94830a4f5764eafee36ba, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.202996 22251 consensus_queue.cc:260] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.203173 22252 tablet_bootstrap.cc:492] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: No bootstrap required, opened a new log
I20260928 14:46:25.203302 22252 ts_tablet_manager.cc:1397] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 14:46:25.203359 22251 ts_tablet_manager.cc:1428] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:46:25.204243 22252 raft_consensus.cc:348] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.204376 22252 raft_consensus.cc:374] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.204422 22252 raft_consensus.cc:729] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59656799d778444c877497c40f599ee9, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.204579 22252 consensus_queue.cc:260] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.204885 22252 ts_tablet_manager.cc:1428] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:46:25.418254 22258 raft_consensus.cc:482] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:25.418468 22258 raft_consensus.cc:504] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.419955 22258 leader_election.cc:288] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:25.424537 22248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.420423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:55105 (local address 127.13.223.67:39649)
0928 14:46:25.420562 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.420567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.420886 (+ 319us) server_negotiation.cc:407] Connection header received
0928 14:46:25.421049 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.421082 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.421150 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.421237 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:25.421676 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.422162 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.423468 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.423930 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.424062 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.424131 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.424309 (+ 178us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.424390 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:46:25.424571 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.420367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:38595 (local address 127.13.223.66:38703)
0928 14:46:25.420510 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.420515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.420694 (+ 179us) server_negotiation.cc:407] Connection header received
0928 14:46:25.420930 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.420936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.421002 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.421074 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:25.421568 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.422189 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.423505 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.423948 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.424093 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.424149 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.424369 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.424454 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:46:25.424582 22281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.420174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:38703 (local address 127.13.223.65:38595)
0928 14:46:25.420576 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.420593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.420715 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.421124 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.421131 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.421150 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:25.421421 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.421431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.422387 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.422390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.423340 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.423350 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.424124 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.424128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.424213 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.424230 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.424335 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.424403 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"mutex_wait_us":5,"thread_start_us":124,"threads_started":1}
I20260928 14:46:25.424590 22282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.420304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:39649 (local address 127.13.223.65:55105)
0928 14:46:25.420764 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.420781 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.420897 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.421348 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.421351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.421376 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:25.421552 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.421561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.422294 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.422297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.423331 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.423341 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.424063 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.424066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.424171 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.424197 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.424265 (+ 68us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.424360 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":47,"thread_start_us":105,"threads_started":1}
I20260928 14:46:25.425236 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835167d4e24ad595cede1810f2156f" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba" is_pre_election: true
I20260928 14:46:25.425325 22037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835167d4e24ad595cede1810f2156f" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86abce5dca142a8842a3a5358d0a558" is_pre_election: true
I20260928 14:46:25.425424 22172 raft_consensus.cc:2454] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 0.
I20260928 14:46:25.425592 22037 raft_consensus.cc:2454] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 0.
I20260928 14:46:25.425884 21831 leader_election.cc:302] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, 59656799d778444c877497c40f599ee9; no voters:
I20260928 14:46:25.426187 22258 raft_consensus.cc:2781] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:25.426251 22258 raft_consensus.cc:482] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:25.426281 22258 raft_consensus.cc:3037] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.427560 22258 raft_consensus.cc:504] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.428030 22258 leader_election.cc:288] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 election: Requested vote from peers c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:25.428471 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835167d4e24ad595cede1810f2156f" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba"
I20260928 14:46:25.428519 22037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6835167d4e24ad595cede1810f2156f" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86abce5dca142a8842a3a5358d0a558"
I20260928 14:46:25.428625 22172 raft_consensus.cc:3037] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.428660 22037 raft_consensus.cc:3037] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.429602 22172 raft_consensus.cc:2454] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 1.
I20260928 14:46:25.429684 22037 raft_consensus.cc:2454] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 1.
I20260928 14:46:25.429975 21831 leader_election.cc:302] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, 59656799d778444c877497c40f599ee9; no voters:
I20260928 14:46:25.430184 22258 raft_consensus.cc:2781] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:25.430433 22258 raft_consensus.cc:686] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 59656799d778444c877497c40f599ee9, State: Running, Role: LEADER
I20260928 14:46:25.430735 22258 consensus_queue.cc:237] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.433432 21738 catalog_manager.cc:5495] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59656799d778444c877497c40f599ee9 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "59656799d778444c877497c40f599ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
Sep 28 14:46:25 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606783, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.126@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:25.523459 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.512784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35795 (local address 127.0.0.1:51550)
0928 14:46:25.513190 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.513212 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.513319 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.513795 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.513798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.514217 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:25.514331 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.514338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.515294 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.515298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.515911 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.515923 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.516509 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.516513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.516593 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.517284 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:25.517297 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:25.519034 (+ 1737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:25.521288 (+ 2254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:25.521302 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:25.521307 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:25.521699 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:25.522066 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:25.522069 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:25.522071 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:25.522189 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:25.522675 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:25.522682 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:25.522964 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.523117 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.523298 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":174,"threads_started":1}
I20260928 14:46:25.523653 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.513002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.13.223.126:35795)
0928 14:46:25.513293 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.513300 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.513316 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:25.513471 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.513473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.513521 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.513607 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:25.514490 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.515175 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.516070 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.516357 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.516463 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.519205 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:25.519209 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:25.519211 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:25.519238 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:25.521144 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:25.521838 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:25.521842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:25.521843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:25.521947 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:25.522286 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:25.522289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:25.522290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:25.522515 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:25.522666 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.523212 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.523411 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":117,"threads_started":1}
I20260928 14:46:25.528880 21738 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51550:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 14:46:25.529691 21738 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:25.536391 21877 tablet_service.cc:1467] Processing CreateTablet for tablet b6d099c0368c4bfdaadeccd02204c376 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ce1446c43994699b6f0cbc2cdd5430f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:46:25.536391 22017 tablet_service.cc:1467] Processing CreateTablet for tablet b6d099c0368c4bfdaadeccd02204c376 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ce1446c43994699b6f0cbc2cdd5430f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:46:25.536464 22157 tablet_service.cc:1467] Processing CreateTablet for tablet b6d099c0368c4bfdaadeccd02204c376 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=4ce1446c43994699b6f0cbc2cdd5430f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 14:46:25.536863 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d099c0368c4bfdaadeccd02204c376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.536863 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d099c0368c4bfdaadeccd02204c376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.536914 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6d099c0368c4bfdaadeccd02204c376. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.540175 22252 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: Bootstrap starting.
I20260928 14:46:25.540195 22251 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: Bootstrap starting.
I20260928 14:46:25.541189 22251 tablet_bootstrap.cc:654] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.541188 22252 tablet_bootstrap.cc:654] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.541978 22254 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: Bootstrap starting.
I20260928 14:46:25.542752 22251 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: No bootstrap required, opened a new log
I20260928 14:46:25.542855 22251 ts_tablet_manager.cc:1397] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:46:25.543071 22254 tablet_bootstrap.cc:654] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.543139 22252 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: No bootstrap required, opened a new log
I20260928 14:46:25.543246 22252 ts_tablet_manager.cc:1397] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:25.543414 22251 raft_consensus.cc:348] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.543568 22251 raft_consensus.cc:374] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.543613 22251 raft_consensus.cc:729] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba036f00c94830a4f5764eafee36ba, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.543762 22251 consensus_queue.cc:260] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.544153 22251 ts_tablet_manager.cc:1428] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:46:25.544091 22257 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.544348 22257 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.544462 22254 tablet_bootstrap.cc:492] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: No bootstrap required, opened a new log
I20260928 14:46:25.543840 22252 raft_consensus.cc:348] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.544557 22254 ts_tablet_manager.cc:1397] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:25.544574 22252 raft_consensus.cc:374] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.544631 22252 raft_consensus.cc:729] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59656799d778444c877497c40f599ee9, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.544782 22252 consensus_queue.cc:260] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.545086 22252 ts_tablet_manager.cc:1428] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:46:25.545065 22254 raft_consensus.cc:348] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.545199 22254 raft_consensus.cc:374] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.545248 22254 raft_consensus.cc:729] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.545404 22254 consensus_queue.cc:260] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.545708 22254 ts_tablet_manager.cc:1428] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:46:25.545996 22258 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.546152 22258 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.546361 22270 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.546543 22270 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.689275 22270 raft_consensus.cc:482] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:25.689508 22270 raft_consensus.cc:504] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.690080 22270 leader_election.cc:288] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59656799d778444c877497c40f599ee9 (127.13.223.65:44515), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:25.690675 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d099c0368c4bfdaadeccd02204c376" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba" is_pre_election: true
I20260928 14:46:25.690675 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d099c0368c4bfdaadeccd02204c376" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9" is_pre_election: true
I20260928 14:46:25.690855 22172 raft_consensus.cc:2454] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:25.690970 21897 raft_consensus.cc:2454] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:25.691293 21971 leader_election.cc:302] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:25.691512 22270 raft_consensus.cc:2781] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:25.691599 22270 raft_consensus.cc:482] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:25.691660 22270 raft_consensus.cc:3037] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.692866 22270 raft_consensus.cc:504] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.693641 22270 leader_election.cc:288] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Requested vote from peers 59656799d778444c877497c40f599ee9 (127.13.223.65:44515), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:25.694011 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d099c0368c4bfdaadeccd02204c376" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9"
I20260928 14:46:25.694161 21897 raft_consensus.cc:3037] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.694115 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6d099c0368c4bfdaadeccd02204c376" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba"
I20260928 14:46:25.694253 22172 raft_consensus.cc:3037] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:25.695497 21897 raft_consensus.cc:2454] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:25.695497 22172 raft_consensus.cc:2454] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:25.695833 21970 leader_election.cc:302] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59656799d778444c877497c40f599ee9, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:25.696071 22270 raft_consensus.cc:2781] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:25.696280 22270 raft_consensus.cc:686] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [term 1 LEADER]: Becoming Leader. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Running, Role: LEADER
I20260928 14:46:25.696455 22270 consensus_queue.cc:237] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.697227 22295 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: TxnStatusTablet state changed. Reason: New leader c86abce5dca142a8842a3a5358d0a558. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.697353 22295 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:46:25.697722 22297 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 14:46:25.697911 22297 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 14:46:25.698463 21740 catalog_manager.cc:5495] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86abce5dca142a8842a3a5358d0a558 (127.13.223.66). New cstate: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:25.743196 22290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.738539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:38703 (local address 127.0.0.1:39226)
0928 14:46:25.738767 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.738787 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.738925 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.739326 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.739329 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.739337 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:46:25.739460 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.739467 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.740640 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.740643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.741373 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.741384 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.742149 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.742152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.742252 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.742322 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.742735 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.742757 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.742940 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.743043 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260928 14:46:25.743201 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.738653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39226 (local address 127.13.223.66:38703)
0928 14:46:25.738798 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.738803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.738871 (+ 68us) server_negotiation.cc:407] Connection header received
0928 14:46:25.739079 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.739083 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.739151 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.739253 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:25.739597 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.740506 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.741520 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.741941 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.742073 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.742482 (+ 409us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.742626 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.742709 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.742975 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.743048 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:46:25.746711 21897 raft_consensus.cc:1261] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.746747 22172 raft_consensus.cc:1261] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.747256 22295 consensus_queue.cc:1035] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.747555 22270 consensus_queue.cc:1035] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.749013 22257 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: TxnStatusTablet state changed. Reason: New leader c86abce5dca142a8842a3a5358d0a558. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.749146 22257 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.749135 22258 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: TxnStatusTablet state changed. Reason: New leader c86abce5dca142a8842a3a5358d0a558. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.749258 22258 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.752324 22300 mvcc.cc:205] Tried to move back new op lower bound from 7334325394413334528 to 7334325394214445056. Current Snapshot: MvccSnapshot[applied={T|T < 7334325394413334528}]
I20260928 14:46:25.752471 22270 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.752597 22270 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:46:25.752876 22258 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.753016 22258 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 59656799d778444c877497c40f599ee9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.753803 22295 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.754014 22295 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P c86abce5dca142a8842a3a5358d0a558: This TxnStatusTablet replica's current role is: LEADER
I20260928 14:46:25.755119 22257 tablet_replica.cc:439] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } } }
I20260928 14:46:25.755246 22257 tablet_replica.cc:442] T b6d099c0368c4bfdaadeccd02204c376 P 30ba036f00c94830a4f5764eafee36ba: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 14:46:25.755430 22302 mvcc.cc:205] Tried to move back new op lower bound from 7334325394413334528 to 7334325394214445056. Current Snapshot: MvccSnapshot[applied={T|T < 7334325394413334528}]
I20260928 14:46:25.767630 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.762333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51960 (local address 127.13.223.65:44515)
0928 14:46:25.762714 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.762720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.762737 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:25.762837 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.762841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.762928 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.763079 (+ 151us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:25.763351 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.764306 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.765417 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.765828 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.765961 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.766341 (+ 380us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.767072 (+ 731us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.767203 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.767314 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.767389 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
I20260928 14:46:25.780241 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.776370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:33815 (local address 127.13.223.66:38703)
0928 14:46:25.776545 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.776551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.776806 (+ 255us) server_negotiation.cc:407] Connection header received
0928 14:46:25.776974 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.776978 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.777037 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.777122 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:25.777415 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.777937 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.778965 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.779214 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.779325 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.779671 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.779794 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.779887 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.780011 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.780073 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 14:46:25.780701 22313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.776285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:38703 (local address 127.13.223.65:33815)
0928 14:46:25.776700 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.776713 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.776849 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.777135 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.777138 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.777146 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:46:25.777297 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.777303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.778057 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.778060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.778830 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.778841 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.779343 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.779370 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.779468 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.779519 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.779902 (+ 383us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.779910 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.779997 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.780090 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":124,"threads_started":1}
I20260928 14:46:25.789032 22172 raft_consensus.cc:1261] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Refusing update from remote peer 59656799d778444c877497c40f599ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.789188 22037 raft_consensus.cc:1261] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Refusing update from remote peer 59656799d778444c877497c40f599ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:25.789598 22258 consensus_queue.cc:1035] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.789916 22283 consensus_queue.cc:1035] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:25.818513 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.813316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:39685 (local address 127.13.223.65:44515)
0928 14:46:25.813502 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.813508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.813644 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:46:25.813744 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.813747 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.813809 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.813914 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:25.814374 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.815236 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.817049 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.817301 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.817445 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.817777 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.817871 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.817968 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.818287 (+ 319us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.818361 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:46:25.818986 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.813098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44515 (local address 127.13.223.66:39685)
0928 14:46:25.813536 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:25.813553 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:25.813659 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:25.813995 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:25.813997 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:25.814006 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:46:25.814114 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.814119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.815683 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.815687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.816814 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:25.816827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.817451 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.817455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:25.817555 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.817623 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:46:25.818154 (+ 531us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:46:25.818162 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:46:25.818714 (+ 552us) client_negotiation.cc:240] Negotiation successful
0928 14:46:25.818836 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":142,"threads_started":1}
I20260928 14:46:25.861202 21740 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:25.861424 21740 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:25.864636 21740 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Sending DeleteTablet for 3 replicas of tablet e6835167d4e24ad595cede1810f2156f
I20260928 14:46:25.865474 22157 tablet_service.cc:1514] Processing DeleteTablet for tablet e6835167d4e24ad595cede1810f2156f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49992
I20260928 14:46:25.865477 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet e6835167d4e24ad595cede1810f2156f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39208
I20260928 14:46:25.866273 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet e6835167d4e24ad595cede1810f2156f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:25 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51956
I20260928 14:46:25.866993 22326 tablet_replica.cc:330] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: stopping tablet replica
I20260928 14:46:25.867304 22326 raft_consensus.cc:2229] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:25.867758 22326 raft_consensus.cc:2258] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:25.868695 22326 ts_tablet_manager.cc:1905] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.870769 22325 tablet_replica.cc:330] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: stopping tablet replica
I20260928 14:46:25.870976 22325 raft_consensus.cc:2229] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:25.871142 22325 raft_consensus.cc:2258] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:25.872758 22326 ts_tablet_manager.cc:1918] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:46:25.872931 22326 log.cc:1196] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/e6835167d4e24ad595cede1810f2156f
I20260928 14:46:25.873344 22326 ts_tablet_manager.cc:1939] T e6835167d4e24ad595cede1810f2156f P 59656799d778444c877497c40f599ee9: Deleting consensus metadata
I20260928 14:46:25.874317 21726 catalog_manager.cc:4841] TS 59656799d778444c877497c40f599ee9 (127.13.223.65:44515): tablet e6835167d4e24ad595cede1810f2156f (table test-table [id=642dc9d034c144fca4bfadcbe5f14167]) successfully deleted
I20260928 14:46:25.878860 22324 tablet_replica.cc:330] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: stopping tablet replica
I20260928 14:46:25.879163 22324 raft_consensus.cc:2229] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:25.879297 22325 ts_tablet_manager.cc:1905] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.879688 22324 raft_consensus.cc:2258] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:25.880722 22324 ts_tablet_manager.cc:1905] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:25.883494 22325 ts_tablet_manager.cc:1918] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:46:25.883615 22325 log.cc:1196] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/e6835167d4e24ad595cede1810f2156f
I20260928 14:46:25.883919 22325 ts_tablet_manager.cc:1939] T e6835167d4e24ad595cede1810f2156f P c86abce5dca142a8842a3a5358d0a558: Deleting consensus metadata
Sep 28 14:46:25 dist-test-slave-gkhx krb5kdc[21695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260928 14:46:25.884454 22324 ts_tablet_manager.cc:1918] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:46:25.884624 22324 log.cc:1196] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/e6835167d4e24ad595cede1810f2156f
I20260928 14:46:25.884799 21726 catalog_manager.cc:4841] TS c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703): tablet e6835167d4e24ad595cede1810f2156f (table test-table [id=642dc9d034c144fca4bfadcbe5f14167]) successfully deleted
I20260928 14:46:25.885030 22324 ts_tablet_manager.cc:1939] T e6835167d4e24ad595cede1810f2156f P 30ba036f00c94830a4f5764eafee36ba: Deleting consensus metadata
I20260928 14:46:25.886154 21727 catalog_manager.cc:4841] TS 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649): tablet e6835167d4e24ad595cede1810f2156f (table test-table [id=642dc9d034c144fca4bfadcbe5f14167]) successfully deleted
Sep 28 14:46:25 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:25 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:25.903021 22291 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:25.893968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51558 (local address 127.13.223.126:35795)
0928 14:46:25.894134 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.894140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.894425 (+ 285us) server_negotiation.cc:407] Connection header received
0928 14:46:25.894649 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.894653 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.894714 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.894807 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:25.895877 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.896753 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.898145 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.898544 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.898682 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.902844 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51558: BlockingRecv error: recv got EOF from 127.0.0.1:51558 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:46:25 dist-test-slave-gkhx krb5kdc[21695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM
I20260928 14:46:25.917680 22291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:25.906939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51572 (local address 127.13.223.126:35795)
0928 14:46:25.907065 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:25.907070 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:25.907268 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:46:25.907493 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:25.907496 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:25.907553 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:25.907665 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:25.908359 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.908932 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.909875 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:25.910141 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:25.910260 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:25.913694 (+ 3434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:25.913699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:25.913701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:25.913750 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:25.915690 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:25.916260 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:25.916265 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:25.916267 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:25.916367 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:25.916707 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:25.916711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:25.916713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:25.916938 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:25.917086 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:25.917360 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:46:25.917504 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 14:46:25.921100 21740 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:25.921726 21740 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:25.928398 21877 tablet_service.cc:1467] Processing CreateTablet for tablet e066cdcac6dd425399a6347ae640a233 (DEFAULT_TABLE table=test-table [id=76fee9d72a98476c812c8cc98cf8b154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.928736 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066cdcac6dd425399a6347ae640a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.931015 22017 tablet_service.cc:1467] Processing CreateTablet for tablet e066cdcac6dd425399a6347ae640a233 (DEFAULT_TABLE table=test-table [id=76fee9d72a98476c812c8cc98cf8b154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.931324 22252 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Bootstrap starting.
I20260928 14:46:25.931348 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066cdcac6dd425399a6347ae640a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.932327 22252 tablet_bootstrap.cc:654] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.933634 22254 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Bootstrap starting.
I20260928 14:46:25.933736 22252 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: No bootstrap required, opened a new log
I20260928 14:46:25.933830 22252 ts_tablet_manager.cc:1397] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:25.934088 22157 tablet_service.cc:1467] Processing CreateTablet for tablet e066cdcac6dd425399a6347ae640a233 (DEFAULT_TABLE table=test-table [id=76fee9d72a98476c812c8cc98cf8b154]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:25.934370 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e066cdcac6dd425399a6347ae640a233. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:25.934304 22252 raft_consensus.cc:348] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.934425 22252 raft_consensus.cc:374] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.934456 22252 raft_consensus.cc:729] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59656799d778444c877497c40f599ee9, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.934525 22254 tablet_bootstrap.cc:654] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.934604 22252 consensus_queue.cc:260] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.934898 22252 ts_tablet_manager.cc:1428] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:46:25.937319 22254 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: No bootstrap required, opened a new log
I20260928 14:46:25.937433 22254 ts_tablet_manager.cc:1397] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260928 14:46:25.937939 22254 raft_consensus.cc:348] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.938078 22254 raft_consensus.cc:374] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.938122 22254 raft_consensus.cc:729] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.938282 22254 consensus_queue.cc:260] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.938694 22254 ts_tablet_manager.cc:1428] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 14:46:25.939077 22251 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Bootstrap starting.
I20260928 14:46:25.940129 22251 tablet_bootstrap.cc:654] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:25.942058 22251 tablet_bootstrap.cc:492] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: No bootstrap required, opened a new log
I20260928 14:46:25.942169 22251 ts_tablet_manager.cc:1397] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 14:46:25.942664 22251 raft_consensus.cc:348] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.942772 22251 raft_consensus.cc:374] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:25.942794 22251 raft_consensus.cc:729] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba036f00c94830a4f5764eafee36ba, State: Initialized, Role: FOLLOWER
I20260928 14:46:25.942955 22251 consensus_queue.cc:260] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:25.943233 22251 ts_tablet_manager.cc:1428] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 14:46:26.077019 22258 raft_consensus.cc:482] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:26.077188 22258 raft_consensus.cc:504] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.077618 22258 leader_election.cc:288] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:26.078244 22037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066cdcac6dd425399a6347ae640a233" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86abce5dca142a8842a3a5358d0a558" is_pre_election: true
I20260928 14:46:26.078250 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066cdcac6dd425399a6347ae640a233" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba" is_pre_election: true
I20260928 14:46:26.078433 22037 raft_consensus.cc:2454] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 0.
I20260928 14:46:26.078500 22172 raft_consensus.cc:2454] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 0.
I20260928 14:46:26.078783 21830 leader_election.cc:302] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59656799d778444c877497c40f599ee9, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:26.078980 22258 raft_consensus.cc:2781] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:26.079044 22258 raft_consensus.cc:482] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:26.079085 22258 raft_consensus.cc:3037] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.080242 22258 raft_consensus.cc:504] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.080683 22258 leader_election.cc:288] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 election: Requested vote from peers c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:26.081161 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066cdcac6dd425399a6347ae640a233" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba"
I20260928 14:46:26.081291 22037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e066cdcac6dd425399a6347ae640a233" candidate_uuid: "59656799d778444c877497c40f599ee9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c86abce5dca142a8842a3a5358d0a558"
I20260928 14:46:26.081396 22172 raft_consensus.cc:3037] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.081446 22037 raft_consensus.cc:3037] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.082450 22037 raft_consensus.cc:2454] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 1.
I20260928 14:46:26.082492 22172 raft_consensus.cc:2454] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59656799d778444c877497c40f599ee9 in term 1.
I20260928 14:46:26.082888 21831 leader_election.cc:302] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, 59656799d778444c877497c40f599ee9, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:26.083163 22258 raft_consensus.cc:2781] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:26.083284 22258 raft_consensus.cc:686] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 1 LEADER]: Becoming Leader. State: Replica: 59656799d778444c877497c40f599ee9, State: Running, Role: LEADER
I20260928 14:46:26.083444 22258 consensus_queue.cc:237] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.085548 21740 catalog_manager.cc:5495] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59656799d778444c877497c40f599ee9 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "59656799d778444c877497c40f599ee9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:26.097930 22310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:26.094116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51966 (local address 127.13.223.65:44515)
0928 14:46:26.094299 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:26.094303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:26.094318 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:46:26.094370 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:26.094373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:26.094416 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:26.094521 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:26.094896 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.095508 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.096641 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.096917 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.097039 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:26.097385 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:26.097473 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:26.097543 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:26.097668 (+ 125us) server_negotiation.cc:299] Negotiation successful
0928 14:46:26.097727 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:46:26.100343 22172 raft_consensus.cc:1261] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Refusing update from remote peer 59656799d778444c877497c40f599ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:26.100446 22037 raft_consensus.cc:1261] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Refusing update from remote peer 59656799d778444c877497c40f599ee9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:26.101049 22317 consensus_queue.cc:1035] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:26.101260 22258 consensus_queue.cc:1035] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:26.112106 21735 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:26.112257 21735 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:26.115012 21735 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Sending DeleteTablet for 3 replicas of tablet e066cdcac6dd425399a6347ae640a233
I20260928 14:46:26.116117 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet e066cdcac6dd425399a6347ae640a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51956
I20260928 14:46:26.116286 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet e066cdcac6dd425399a6347ae640a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39208
I20260928 14:46:26.116228 22157 tablet_service.cc:1514] Processing DeleteTablet for tablet e066cdcac6dd425399a6347ae640a233 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49992
I20260928 14:46:26.116539 22325 tablet_replica.cc:330] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: stopping tablet replica
I20260928 14:46:26.116528 22326 tablet_replica.cc:330] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: stopping tablet replica
I20260928 14:46:26.116693 22325 raft_consensus.cc:2229] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:26.116720 22326 raft_consensus.cc:2229] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:26.116863 22325 raft_consensus.cc:2258] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.116984 22326 raft_consensus.cc:2258] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.117198 22324 tablet_replica.cc:330] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: stopping tablet replica
I20260928 14:46:26.117341 22324 raft_consensus.cc:2229] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:26.117519 22324 raft_consensus.cc:2258] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.117628 21735 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 14:46:26.117916 22325 ts_tablet_manager.cc:1905] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:26.118319 22324 ts_tablet_manager.cc:1905] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:46:26.118321 21735 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:26.121124 22324 ts_tablet_manager.cc:1918] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:26.121147 22325 ts_tablet_manager.cc:1918] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:26.121230 22324 log.cc:1196] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/e066cdcac6dd425399a6347ae640a233
I20260928 14:46:26.121259 22325 log.cc:1196] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/e066cdcac6dd425399a6347ae640a233
I20260928 14:46:26.121685 22325 ts_tablet_manager.cc:1939] T e066cdcac6dd425399a6347ae640a233 P c86abce5dca142a8842a3a5358d0a558: Deleting consensus metadata
I20260928 14:46:26.121682 22324 ts_tablet_manager.cc:1939] T e066cdcac6dd425399a6347ae640a233 P 30ba036f00c94830a4f5764eafee36ba: Deleting consensus metadata
I20260928 14:46:26.121922 22326 ts_tablet_manager.cc:1905] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:26.122587 21726 catalog_manager.cc:4841] TS c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703): tablet e066cdcac6dd425399a6347ae640a233 (table test-table [id=76fee9d72a98476c812c8cc98cf8b154]) successfully deleted
I20260928 14:46:26.122692 21727 catalog_manager.cc:4841] TS 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649): tablet e066cdcac6dd425399a6347ae640a233 (table test-table [id=76fee9d72a98476c812c8cc98cf8b154]) successfully deleted
I20260928 14:46:26.124128 22326 ts_tablet_manager.cc:1918] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:26.124198 22326 log.cc:1196] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/e066cdcac6dd425399a6347ae640a233
I20260928 14:46:26.124450 22326 ts_tablet_manager.cc:1939] T e066cdcac6dd425399a6347ae640a233 P 59656799d778444c877497c40f599ee9: Deleting consensus metadata
I20260928 14:46:26.125520 21726 catalog_manager.cc:4841] TS 59656799d778444c877497c40f599ee9 (127.13.223.65:44515): tablet e066cdcac6dd425399a6347ae640a233 (table test-table [id=76fee9d72a98476c812c8cc98cf8b154]) successfully deleted
I20260928 14:46:26.128767 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 21c1827947c744498012b20f6d1396f8 (DEFAULT_TABLE table=test-table [id=7f89746504f345d0abd76e4f247ee00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:26.128922 22157 tablet_service.cc:1467] Processing CreateTablet for tablet 21c1827947c744498012b20f6d1396f8 (DEFAULT_TABLE table=test-table [id=7f89746504f345d0abd76e4f247ee00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:26.129031 22017 tablet_service.cc:1467] Processing CreateTablet for tablet 21c1827947c744498012b20f6d1396f8 (DEFAULT_TABLE table=test-table [id=7f89746504f345d0abd76e4f247ee00f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:26.129165 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c1827947c744498012b20f6d1396f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:26.129245 22157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c1827947c744498012b20f6d1396f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:26.129382 22017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 21c1827947c744498012b20f6d1396f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:26.131717 22254 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Bootstrap starting.
I20260928 14:46:26.131855 22252 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: Bootstrap starting.
I20260928 14:46:26.131848 22251 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Bootstrap starting.
I20260928 14:46:26.132565 22254 tablet_bootstrap.cc:654] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:26.132833 22252 tablet_bootstrap.cc:654] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:26.133316 22251 tablet_bootstrap.cc:654] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:26.133711 22254 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: No bootstrap required, opened a new log
I20260928 14:46:26.133810 22254 ts_tablet_manager.cc:1397] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 14:46:26.134315 22254 raft_consensus.cc:348] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.134442 22254 raft_consensus.cc:374] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:26.134480 22254 raft_consensus.cc:729] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Initialized, Role: FOLLOWER
I20260928 14:46:26.134630 22254 consensus_queue.cc:260] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.134730 22251 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: No bootstrap required, opened a new log
I20260928 14:46:26.134829 22251 ts_tablet_manager.cc:1397] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 14:46:26.134954 22254 ts_tablet_manager.cc:1428] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 14:46:26.134996 22252 tablet_bootstrap.cc:492] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: No bootstrap required, opened a new log
I20260928 14:46:26.135077 22252 ts_tablet_manager.cc:1397] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 14:46:26.135366 22251 raft_consensus.cc:348] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.135502 22251 raft_consensus.cc:374] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:26.135555 22251 raft_consensus.cc:729] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba036f00c94830a4f5764eafee36ba, State: Initialized, Role: FOLLOWER
I20260928 14:46:26.135516 22252 raft_consensus.cc:348] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.135658 22252 raft_consensus.cc:374] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:26.135695 22252 raft_consensus.cc:729] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59656799d778444c877497c40f599ee9, State: Initialized, Role: FOLLOWER
I20260928 14:46:26.135706 22251 consensus_queue.cc:260] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.135856 22252 consensus_queue.cc:260] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.136024 22251 ts_tablet_manager.cc:1428] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:46:26.136132 22252 ts_tablet_manager.cc:1428] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:46:26.184577 22295 raft_consensus.cc:482] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:26.184748 22295 raft_consensus.cc:504] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.185320 22295 leader_election.cc:288] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59656799d778444c877497c40f599ee9 (127.13.223.65:44515), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:26.185950 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c1827947c744498012b20f6d1396f8" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba" is_pre_election: true
I20260928 14:46:26.185999 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c1827947c744498012b20f6d1396f8" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9" is_pre_election: true
I20260928 14:46:26.186156 22172 raft_consensus.cc:2454] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:26.186180 21897 raft_consensus.cc:2454] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 0.
I20260928 14:46:26.186625 21971 leader_election.cc:302] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:26.186969 22295 raft_consensus.cc:2781] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:26.187047 22295 raft_consensus.cc:482] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:26.187098 22295 raft_consensus.cc:3037] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.188613 22295 raft_consensus.cc:504] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.189144 22295 leader_election.cc:288] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Requested vote from peers 59656799d778444c877497c40f599ee9 (127.13.223.65:44515), 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649)
I20260928 14:46:26.189636 22172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c1827947c744498012b20f6d1396f8" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "30ba036f00c94830a4f5764eafee36ba"
I20260928 14:46:26.189648 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "21c1827947c744498012b20f6d1396f8" candidate_uuid: "c86abce5dca142a8842a3a5358d0a558" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59656799d778444c877497c40f599ee9"
I20260928 14:46:26.189774 21897 raft_consensus.cc:3037] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.189802 22172 raft_consensus.cc:3037] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:26.190872 22172 raft_consensus.cc:2454] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:26.190966 21897 raft_consensus.cc:2454] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c86abce5dca142a8842a3a5358d0a558 in term 1.
I20260928 14:46:26.191294 21971 leader_election.cc:302] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 30ba036f00c94830a4f5764eafee36ba, c86abce5dca142a8842a3a5358d0a558; no voters:
I20260928 14:46:26.191514 22295 raft_consensus.cc:2781] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:26.191627 22295 raft_consensus.cc:686] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 1 LEADER]: Becoming Leader. State: Replica: c86abce5dca142a8842a3a5358d0a558, State: Running, Role: LEADER
I20260928 14:46:26.191773 22295 consensus_queue.cc:237] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } }
I20260928 14:46:26.194290 21738 catalog_manager.cc:5495] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 reported cstate change: term changed from 0 to 1, leader changed from <none> to c86abce5dca142a8842a3a5358d0a558 (127.13.223.66). New cstate: current_term: 1 leader_uuid: "c86abce5dca142a8842a3a5358d0a558" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c86abce5dca142a8842a3a5358d0a558" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 38703 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:26.216928 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:26.213271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39228 (local address 127.13.223.66:38703)
0928 14:46:26.213413 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:26.213419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:26.213436 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:26.213562 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:26.213567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:26.213626 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:26.213717 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:26.214082 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.214745 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.215701 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.215966 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.216096 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:26.216401 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:26.216491 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:26.216585 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:26.216690 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:46:26.216748 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:46:26.228111 22247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:26.224185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:58267 (local address 127.13.223.66:38703)
0928 14:46:26.224304 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:26.224308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:26.224322 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:46:26.224522 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:26.224524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:26.224569 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:26.224649 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:26.224944 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.225580 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.226734 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.227044 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.227139 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:26.227491 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:26.227576 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:26.227651 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:26.227840 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:46:26.227974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:46:26.228161 22321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:26.224038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:38703 (local address 127.13.223.66:58267)
0928 14:46:26.224221 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:26.224239 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:26.224336 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:26.224679 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:26.224681 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:26.224689 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 14:46:26.224815 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:26.224820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.225723 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.225726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:26.226484 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:26.226494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:26.227185 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:26.227188 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:26.227301 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:26.227334 (+ 33us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 14:46:26.227690 (+ 356us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 14:46:26.227697 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:46:26.227769 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:46:26.227949 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:46:26.235623 21897 raft_consensus.cc:1261] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:26.235695 22172 raft_consensus.cc:1261] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Refusing update from remote peer c86abce5dca142a8842a3a5358d0a558: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:26.236104 22319 consensus_queue.cc:1035] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59656799d778444c877497c40f599ee9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:26.236269 22295 consensus_queue.cc:1035] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [LEADER]: Connected to new peer: Peer: permanent_uuid: "30ba036f00c94830a4f5764eafee36ba" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:26.276129 21738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:26.276292 21738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:26.278507 21738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 869a44c876e642e686e6a83b7cc32558: Sending DeleteTablet for 3 replicas of tablet 21c1827947c744498012b20f6d1396f8
I20260928 14:46:26.279291 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c1827947c744498012b20f6d1396f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51956
I20260928 14:46:26.279371 22017 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c1827947c744498012b20f6d1396f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39208
I20260928 14:46:26.279428 22157 tablet_service.cc:1514] Processing DeleteTablet for tablet 21c1827947c744498012b20f6d1396f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:26 UTC) from {username='oryx', principal='oryx/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49992
I20260928 14:46:26.279515 22326 tablet_replica.cc:330] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: stopping tablet replica
I20260928 14:46:26.279572 22325 tablet_replica.cc:330] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: stopping tablet replica
I20260928 14:46:26.279637 22324 tablet_replica.cc:330] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: stopping tablet replica
I20260928 14:46:26.279640 22326 raft_consensus.cc:2229] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:26.279719 22325 raft_consensus.cc:2229] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:26.279763 22324 raft_consensus.cc:2229] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:26.279759 22326 raft_consensus.cc:2258] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.279886 22324 raft_consensus.cc:2258] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.280056 22325 raft_consensus.cc:2258] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:26.280689 22326 ts_tablet_manager.cc:1905] T 21c1827947c744498012b20f6d1396f8 P 59656799d778444c877497c40f599ee9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:26.280797 22325 ts_tablet_manager.cc:1905] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:26.281205 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21812
I20260928 14:46:26.283351 22325 ts_tablet_manager.cc:1918] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:46:26.283422 22325 log.cc:1196] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/21c1827947c744498012b20f6d1396f8
I20260928 14:46:26.283620 22325 ts_tablet_manager.cc:1939] T 21c1827947c744498012b20f6d1396f8 P c86abce5dca142a8842a3a5358d0a558: Deleting consensus metadata
I20260928 14:46:26.284376 22324 ts_tablet_manager.cc:1905] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:26.286339 22324 ts_tablet_manager.cc:1918] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 14:46:26.286412 22324 log.cc:1196] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/21c1827947c744498012b20f6d1396f8
I20260928 14:46:26.286720 22324 ts_tablet_manager.cc:1939] T 21c1827947c744498012b20f6d1396f8 P 30ba036f00c94830a4f5764eafee36ba: Deleting consensus metadata
I20260928 14:46:26.287475 21727 catalog_manager.cc:4841] TS 30ba036f00c94830a4f5764eafee36ba (127.13.223.67:39649): tablet 21c1827947c744498012b20f6d1396f8 (table test-table [id=7f89746504f345d0abd76e4f247ee00f]) successfully deleted
W20260928 14:46:26.291008 21726 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:44515 (error 108)
I20260928 14:46:26.291450 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21952
I20260928 14:46:26.291695 21726 catalog_manager.cc:4841] TS c86abce5dca142a8842a3a5358d0a558 (127.13.223.66:38703): tablet 21c1827947c744498012b20f6d1396f8 (table test-table [id=7f89746504f345d0abd76e4f247ee00f]) successfully deleted
I20260928 14:46:26.293474 22354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:26.292016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44515 (local address 127.0.0.1:51970)
0928 14:46:26.293290 (+ 1274us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:26.293383 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1127,"thread_start_us":54,"threads_started":1}
W20260928 14:46:26.293614 21726 catalog_manager.cc:4584] TS 59656799d778444c877497c40f599ee9 (127.13.223.65:44515): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 21c1827947c744498012b20f6d1396f8: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44515: connect: Connection refused (error 111)
I20260928 14:46:26.299141 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22092
I20260928 14:46:26.305589 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 21708
2026-09-28T14:46:26Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3829 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:26 dist-test-slave-gkhx krb5kdc[22357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:26 dist-test-slave-gkhx krb5kdc[22357](info): set up 2 sockets
Sep 28 14:46:26 dist-test-slave-gkhx krb5kdc[22357](info): commencing operation
krb5kdc: starting...
W20260928 14:46:27.434340 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:27 dist-test-slave-gkhx krb5kdc[22357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606787, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:27.575628 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:45785
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:45785
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:27.697165 22370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:27.697576 22370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:27.697666 22370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:27.702155 22370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:27.702314 22370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:27.702409 22370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:27.702466 22370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:27.702509 22370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:27.708976 22370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:45785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:27.710381 22370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:27.711602 22370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:27.722232 22375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:27.722232 22378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:27.722249 22376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:27.722617 22370 server_base.cc:1034] running on GCE node
I20260928 14:46:27.723555 22370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:27.724877 22370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:27.726133 22370 hybrid_clock.cc:648] HybridClock initialized: now 1790606787726118 us; error 32 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T14:46:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1427 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:27 dist-test-slave-gkhx krb5kdc[22383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:27 dist-test-slave-gkhx krb5kdc[22383](info): set up 2 sockets
Sep 28 14:46:27 dist-test-slave-gkhx krb5kdc[22383](info): commencing operation
krb5kdc: starting...
W20260928 14:46:28.834263 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:28 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:28.975380 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43813
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:29.098759 22396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:29.099148 22396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:29.099244 22396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:29.104189 22396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:29.104326 22396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:29.104367 22396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:29.104411 22396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:29.104449 22396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:29.110836 22396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:29.112159 22396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:29.113346 22396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:29.123450 22404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.123436 22401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:29.123591 22396 server_base.cc:1034] running on GCE node
W20260928 14:46:29.123435 22402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:29.124464 22396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:29.125723 22396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:29.126968 22396 hybrid_clock.cc:648] HybridClock initialized: now 1790606789126943 us; error 42 us; skew 500 ppm
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:29.137548 22396 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:29.138772 22396 webserver.cc:460] Webserver started at http://127.13.223.126:38773/ using document root <none> and password file <none>
I20260928 14:46:29.139405 22396 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:29.139509 22396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:29.139757 22396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:29.142035 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "da31644dc55c4ad58ed453b03b7ec159"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "1acb4482913575a1191a9274ac638baf"
server_key_iv: "8f2c5b32cc1ee994378f921f52b8488c"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.142726 22396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "da31644dc55c4ad58ed453b03b7ec159"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "1acb4482913575a1191a9274ac638baf"
server_key_iv: "8f2c5b32cc1ee994378f921f52b8488c"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.147310 22396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:29.150213 22411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.151752 22396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:29.151928 22396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "da31644dc55c4ad58ed453b03b7ec159"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "1acb4482913575a1191a9274ac638baf"
server_key_iv: "8f2c5b32cc1ee994378f921f52b8488c"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.152086 22396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:29.173002 22396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:29.177253 22396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:29.177561 22396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:29.185961 22396 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:43813
I20260928 14:46:29.185985 22463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:43813 every 8 connection(s)
I20260928 14:46:29.187225 22396 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:29.190501 22464 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:29.193727 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22396
I20260928 14:46:29.193887 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:29.194239 14205 external_mini_cluster.cc:1428] Setting key 30e16ea8bb1f5f8b3330b85e8649a185
I20260928 14:46:29.197824 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Bootstrap starting.
I20260928 14:46:29.200809 22464 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:29.201959 22464 log.cc:824] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:29.204941 22464 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: No bootstrap required, opened a new log
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606788, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:29.209390 22464 raft_consensus.cc:348] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:29.209712 22464 raft_consensus.cc:374] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:29.209816 22464 raft_consensus.cc:729] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da31644dc55c4ad58ed453b03b7ec159, State: Initialized, Role: FOLLOWER
I20260928 14:46:29.210445 22464 consensus_queue.cc:260] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:29.210669 22464 raft_consensus.cc:388] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:29.210764 22464 raft_consensus.cc:482] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:29.210882 22464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:29.212288 22464 raft_consensus.cc:504] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:29.212746 22464 leader_election.cc:302] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da31644dc55c4ad58ed453b03b7ec159; no voters:
I20260928 14:46:29.213168 22464 leader_election.cc:288] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:29.213337 22469 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:29.213594 22469 raft_consensus.cc:686] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 LEADER]: Becoming Leader. State: Replica: da31644dc55c4ad58ed453b03b7ec159, State: Running, Role: LEADER
I20260928 14:46:29.213690 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:29.196016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.13.223.126:43813)
0928 14:46:29.196576 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:29.196593 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:29.196643 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:46:29.197313 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:29.197333 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:29.197708 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:29.198101 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:29.199133 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.200510 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.202055 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.203045 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.203222 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:29.206368 (+ 3146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:29.206377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:29.206397 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:29.206434 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:29.209356 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.210008 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.210015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.210022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.210205 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.210619 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.210622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.210624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.211327 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:29.211574 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:29.212075 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 14:46:29.212481 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":167,"threads_started":1}
I20260928 14:46:29.213982 22469 consensus_queue.cc:237] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:29.214715 22464 sys_catalog.cc:564] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:29.215471 22470 sys_catalog.cc:455] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da31644dc55c4ad58ed453b03b7ec159. Latest consensus state: current_term: 1 leader_uuid: "da31644dc55c4ad58ed453b03b7ec159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } } }
I20260928 14:46:29.215619 22470 sys_catalog.cc:458] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:29.215883 22471 sys_catalog.cc:455] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da31644dc55c4ad58ed453b03b7ec159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } } }
I20260928 14:46:29.215963 22471 sys_catalog.cc:458] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:29.216958 22476 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:29.219866 22476 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:29.225907 22476 catalog_manager.cc:1347] Generated new cluster ID: 37e3d1ae218d48e0919dbfa3e3e327a2
I20260928 14:46:29.225999 22476 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:29.243227 22476 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:29.244791 22476 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:29.254750 22476 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Generated new TSK 0
I20260928 14:46:29.255802 22476 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:46:29.295022 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:29.420455 22490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:29.420989 22490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:29.421114 22490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:29.426076 22490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:29.426219 22490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:29.426339 22490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:29.432825 22490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:29.434154 22490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:29.435362 22490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:29.446542 22498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.446529 22496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.446537 22495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:29.446645 22490 server_base.cc:1034] running on GCE node
I20260928 14:46:29.447335 22490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:29.448086 22490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:29.449265 22490 hybrid_clock.cc:648] HybridClock initialized: now 1790606789449249 us; error 31 us; skew 500 ppm
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:29.452701 22490 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:29.453809 22490 webserver.cc:460] Webserver started at http://127.13.223.65:41421/ using document root <none> and password file <none>
I20260928 14:46:29.454648 22490 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:29.454731 22490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:29.455035 22490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:29.457572 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "0dace2533fd54bb9afeb8674a6193bbd"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "e70bd38713f5d035ffd9a1b1c1f0ea4e"
server_key_iv: "bdd61e26b1f5d8c6063ce4c035e7c5fa"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.458282 22490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "0dace2533fd54bb9afeb8674a6193bbd"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "e70bd38713f5d035ffd9a1b1c1f0ea4e"
server_key_iv: "bdd61e26b1f5d8c6063ce4c035e7c5fa"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.463085 22490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260928 14:46:29.466274 22505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.468003 22490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:29.468361 22490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "0dace2533fd54bb9afeb8674a6193bbd"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "e70bd38713f5d035ffd9a1b1c1f0ea4e"
server_key_iv: "bdd61e26b1f5d8c6063ce4c035e7c5fa"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.468562 22490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:29.483300 22490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:29.487703 22490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:29.487970 22490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:29.488704 22490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:29.489708 22490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:29.489785 22490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.489869 22490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:29.489919 22490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.502204 22490 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:43791
I20260928 14:46:29.502239 22618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:43791 every 8 connection(s)
I20260928 14:46:29.503324 22490 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:29.513717 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22490
I20260928 14:46:29.513967 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:29.514293 14205 external_mini_cluster.cc:1428] Setting key cd21f9ad39dffa1fd5f38b9bebdac064
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:29.521127 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:29.505747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:60705 (local address 127.13.223.126:43813)
0928 14:46:29.505933 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:29.505938 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:29.506991 (+ 1053us) server_negotiation.cc:407] Connection header received
0928 14:46:29.508445 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:29.508448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:29.508502 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:29.508587 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:29.510209 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.510940 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.512260 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.512570 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.512708 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:29.516213 (+ 3505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:29.516219 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:29.516223 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:29.516257 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:29.518199 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.518939 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.518945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.518948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.519069 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.519658 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.519662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.519664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.519926 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:29.520018 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:29.520839 (+ 821us) server_negotiation.cc:299] Negotiation successful
0928 14:46:29.520987 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260928 14:46:29.522600 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:29.506190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:60705)
0928 14:46:29.506758 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:29.506861 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:29.508179 (+ 1318us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:29.508848 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:29.508857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:29.509295 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:29.509985 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:29.510003 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.511103 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.511106 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:29.512098 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:29.512110 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.512753 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.512756 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:29.513003 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:29.513689 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:29.513715 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:29.516021 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:29.518367 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:29.518373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:29.518391 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:29.518803 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:29.519248 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:29.519255 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:29.519259 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:29.519486 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:29.520054 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:29.520065 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:29.520502 (+ 437us) client_negotiation.cc:769] Sending connection context
0928 14:46:29.520740 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 14:46:29.521199 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":139,"threads_started":1}
I20260928 14:46:29.524308 22619 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43813
I20260928 14:46:29.524603 22619 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:29.525170 22619 heartbeater.cc:511] Master 127.13.223.126:43813 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:46:29.527092 22428 ts_manager.cc:194] Registered new tserver with Master: 0dace2533fd54bb9afeb8674a6193bbd (127.13.223.65:43791)
I20260928 14:46:29.528291 22428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:60705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:29.551788 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:29.678247 22624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:29.678676 22624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:29.678759 22624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:29.683555 22624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:29.683645 22624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:29.683712 22624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:29.689517 22624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:29.690706 22624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:29.691752 22624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:46:29.702836 22624 server_base.cc:1034] running on GCE node
W20260928 14:46:29.702697 22632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.702693 22629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.702697 22630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:29.703547 22624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:29.704334 22624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:29.705579 22624 hybrid_clock.cc:648] HybridClock initialized: now 1790606789705566 us; error 29 us; skew 500 ppm
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:29.709225 22624 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:29.710278 22624 webserver.cc:460] Webserver started at http://127.13.223.66:36623/ using document root <none> and password file <none>
I20260928 14:46:29.711019 22624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:29.711126 22624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:29.711385 22624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:29.713624 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "e21a862895fc41269db7c88f61955b1c"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "ca236b7acf393b9594486393e8ba7e0a"
server_key_iv: "80b7e56c09bc2b9aee43cfd49bdc52f0"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.714246 22624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "e21a862895fc41269db7c88f61955b1c"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "ca236b7acf393b9594486393e8ba7e0a"
server_key_iv: "80b7e56c09bc2b9aee43cfd49bdc52f0"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.718708 22624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260928 14:46:29.721549 22639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.723208 22624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:29.723387 22624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "e21a862895fc41269db7c88f61955b1c"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "ca236b7acf393b9594486393e8ba7e0a"
server_key_iv: "80b7e56c09bc2b9aee43cfd49bdc52f0"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.723583 22624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:29.746685 22624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:29.750537 22624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:29.750828 22624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:29.751559 22624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:29.752509 22624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:29.752589 22624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.752683 22624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:29.752738 22624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.763957 22624 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:39155
I20260928 14:46:29.763972 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:39155 every 8 connection(s)
I20260928 14:46:29.765030 22624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:29.770192 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22624
I20260928 14:46:29.770329 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:29.770684 14205 external_mini_cluster.cc:1428] Setting key e0094150e51311bfbe6249b9c2905420
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:46:29.782510 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:29.767090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:33933 (local address 127.13.223.126:43813)
0928 14:46:29.767269 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:29.767274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:29.768265 (+ 991us) server_negotiation.cc:407] Connection header received
0928 14:46:29.769583 (+ 1318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:29.769586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:29.769641 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:29.769726 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:29.771557 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.772410 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.774328 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.774830 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.775006 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:29.777893 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:29.777897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:29.777899 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:29.777929 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:29.779825 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.780571 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.780574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.780576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.780669 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:29.781126 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:29.781128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:29.781130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:29.781389 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:29.781533 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:29.782149 (+ 616us) server_negotiation.cc:299] Negotiation successful
0928 14:46:29.782277 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:46:29.783447 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:29.767525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:33933)
0928 14:46:29.768087 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:29.768138 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:29.769356 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:29.770048 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:29.770060 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:29.770607 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:29.771348 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:29.771370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.772557 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.772561 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:29.774163 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:29.774181 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:29.775083 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:29.775087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:29.775300 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:29.776031 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:29.776049 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:29.777629 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:29.779979 (+ 2350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:29.779988 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:29.780009 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:29.780432 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:29.780788 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:29.780795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:29.780800 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:29.781008 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:29.781552 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:29.781563 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:29.781874 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 14:46:29.782051 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 14:46:29.782410 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":149,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:46:29.784950 22753 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43813
I20260928 14:46:29.785245 22753 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:29.785964 22753 heartbeater.cc:511] Master 127.13.223.126:43813 requested a full tablet report, sending...
I20260928 14:46:29.787380 22428 ts_manager.cc:194] Registered new tserver with Master: e21a862895fc41269db7c88f61955b1c (127.13.223.66:39155)
I20260928 14:46:29.788103 22428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:33933
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:29.807725 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:29.929909 22758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:29.930299 22758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:29.930388 22758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:29.934994 22758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:29.935110 22758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:29.935253 22758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:29.941617 22758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:29.943019 22758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:29.944165 22758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:29.955008 22766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.955006 22763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:29.955006 22764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:29.955706 22758 server_base.cc:1034] running on GCE node
I20260928 14:46:29.956311 22758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:29.957170 22758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:29.958398 22758 hybrid_clock.cc:648] HybridClock initialized: now 1790606789958384 us; error 28 us; skew 500 ppm
Sep 28 14:46:29 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:29.962020 22758 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:29.963348 22758 webserver.cc:460] Webserver started at http://127.13.223.67:43523/ using document root <none> and password file <none>
I20260928 14:46:29.964016 22758 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:29.964123 22758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:29.964355 22758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:29.966765 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "b9fb64f6063a42a48ed930225c68f9f9"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "a8ebf17d25297b50d272b907e0db353d"
server_key_iv: "edbf83c4b65cdf91665faae78545be74"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.967424 22758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "b9fb64f6063a42a48ed930225c68f9f9"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "a8ebf17d25297b50d272b907e0db353d"
server_key_iv: "edbf83c4b65cdf91665faae78545be74"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.971694 22758 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:46:29.974849 22773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:29.976383 22758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:46:29.976550 22758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "b9fb64f6063a42a48ed930225c68f9f9"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "a8ebf17d25297b50d272b907e0db353d"
server_key_iv: "edbf83c4b65cdf91665faae78545be74"
server_key_version: "encryptionkey@0"
I20260928 14:46:29.976688 22758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:29.997906 22758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:30.001991 22758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:30.002264 22758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:30.003021 22758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:30.004012 22758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:30.004093 22758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.004176 22758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:30.004232 22758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.015520 22758 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:36929
I20260928 14:46:30.015543 22886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:36929 every 8 connection(s)
I20260928 14:46:30.016692 22758 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:30.026336 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22758
I20260928 14:46:30.026538 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:30.026925 14205 external_mini_cluster.cc:1428] Setting key 82c1db570f03517af858932dcaf11f17
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606789, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:30.033146 22467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.019098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41777 (local address 127.13.223.126:43813)
0928 14:46:30.019288 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.019294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.020651 (+ 1357us) server_negotiation.cc:407] Connection header received
0928 14:46:30.022316 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.022320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.022374 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.022460 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.024016 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.024603 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.025857 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.026131 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.026217 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.029310 (+ 3093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:30.029315 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:30.029317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:30.029361 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:30.030920 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:30.031446 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:30.031448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:30.031449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:30.031529 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:30.031869 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:30.031872 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:30.031873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:30.032122 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:30.032231 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:30.032862 (+ 631us) server_negotiation.cc:299] Negotiation successful
0928 14:46:30.033013 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:46:30.033990 22889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.019626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:41777)
0928 14:46:30.020428 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.020507 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.022044 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.022726 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.022736 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.023242 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.023811 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.023827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.024754 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.024757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.025719 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.025730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.026317 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.026324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.026597 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.027358 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.027376 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.029123 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:30.031040 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:30.031046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:30.031061 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:30.031336 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:30.031609 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:30.031611 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:30.031613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:30.031753 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:30.032319 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:30.032326 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:30.032603 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 14:46:30.032791 (+ 188us) client_negotiation.cc:240] Negotiation successful
0928 14:46:30.033096 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":221,"threads_started":1}
I20260928 14:46:30.035334 22887 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43813
I20260928 14:46:30.035615 22887 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:30.036219 22887 heartbeater.cc:511] Master 127.13.223.126:43813 requested a full tablet report, sending...
I20260928 14:46:30.037467 22428 ts_manager.cc:194] Registered new tserver with Master: b9fb64f6063a42a48ed930225c68f9f9 (127.13.223.67:36929)
I20260928 14:46:30.038053 22428 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:41777
I20260928 14:46:30.041865 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
Entry for principal oryx/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab.
WARNING: no policy specified for oryx/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
Entry for principal oryx/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab.
WARNING: no policy specified for oryx/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
Entry for principal oryx/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab.
I20260928 14:46:30.135738 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22490
I20260928 14:46:30.142000 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22624
I20260928 14:46:30.148063 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22758
I20260928 14:46:30.153891 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22396
I20260928 14:46:30.161767 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43813
--webserver_interface=127.13.223.126
--webserver_port=38773
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:43813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:30.288103 22896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:30.288421 22896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:30.288465 22896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:30.292737 22896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:30.292829 22896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:30.292850 22896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:30.292868 22896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:30.292889 22896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:30.299129 22896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:43813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=38773
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:30.300406 22896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:30.301539 22896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:30.311770 22901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:30.311758 22902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.311995 22896 server_base.cc:1034] running on GCE node
W20260928 14:46:30.312018 22904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.312840 22896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:30.314194 22896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:30.315416 22896 hybrid_clock.cc:648] HybridClock initialized: now 1790606790315368 us; error 64 us; skew 500 ppm
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606790, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:30.318722 22896 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:30.319916 22896 webserver.cc:460] Webserver started at http://127.13.223.126:38773/ using document root <none> and password file <none>
I20260928 14:46:30.320618 22896 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:30.320758 22896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:30.325505 22896 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:30.328274 22911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.329948 22896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:46:30.330152 22896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "da31644dc55c4ad58ed453b03b7ec159"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "1acb4482913575a1191a9274ac638baf"
server_key_iv: "8f2c5b32cc1ee994378f921f52b8488c"
server_key_version: "encryptionkey@0"
I20260928 14:46:30.330942 22896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:30.357959 22896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:30.362051 22896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:30.362316 22896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:30.370864 22896 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:43813
I20260928 14:46:30.370886 22963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:43813 every 8 connection(s)
I20260928 14:46:30.372196 22896 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:30.377059 22964 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:46:30.379082 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Bootstrap starting.
I20260928 14:46:30.380347 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22896
I20260928 14:46:30.380916 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:43791
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:43813
--webserver_port=41421
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:46:30.398092 22964 log.cc:824] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:30.405983 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:46:30.406446 22964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Bootstrap complete.
I20260928 14:46:30.411108 22964 raft_consensus.cc:348] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:30.411583 22964 raft_consensus.cc:729] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: da31644dc55c4ad58ed453b03b7ec159, State: Initialized, Role: FOLLOWER
I20260928 14:46:30.412210 22964 consensus_queue.cc:260] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:30.412345 22964 raft_consensus.cc:388] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:30.412441 22964 raft_consensus.cc:482] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:30.412542 22964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:46:30.414796 22964 raft_consensus.cc:504] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:30.415190 22964 leader_election.cc:302] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da31644dc55c4ad58ed453b03b7ec159; no voters:
I20260928 14:46:30.415565 22964 leader_election.cc:288] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:46:30.415755 22968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:46:30.415970 22968 raft_consensus.cc:686] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [term 2 LEADER]: Becoming Leader. State: Replica: da31644dc55c4ad58ed453b03b7ec159, State: Running, Role: LEADER
I20260928 14:46:30.416293 22968 consensus_queue.cc:237] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } }
I20260928 14:46:30.416977 22964 sys_catalog.cc:564] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:30.417953 22969 sys_catalog.cc:455] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "da31644dc55c4ad58ed453b03b7ec159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } } }
I20260928 14:46:30.417981 22970 sys_catalog.cc:455] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da31644dc55c4ad58ed453b03b7ec159. Latest consensus state: current_term: 2 leader_uuid: "da31644dc55c4ad58ed453b03b7ec159" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da31644dc55c4ad58ed453b03b7ec159" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43813 } } }
I20260928 14:46:30.418118 22969 sys_catalog.cc:458] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:30.418118 22970 sys_catalog.cc:458] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:30.418526 22977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:30.422190 22977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:30.422991 22977 catalog_manager.cc:1259] Loaded cluster ID: 37e3d1ae218d48e0919dbfa3e3e327a2
I20260928 14:46:30.423054 22977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:30.424871 22977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:30.425544 22977 catalog_manager.cc:5879] T 00000000000000000000000000000000 P da31644dc55c4ad58ed453b03b7ec159: Loaded TSK: 0
I20260928 14:46:30.426848 22977 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:46:30.524730 22966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:30.525053 22966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:30.525105 22966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:30.529775 22966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:30.529856 22966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:30.529922 22966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:30.535837 22966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.65.keytab
--principal=oryx/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:43791
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=41421
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:30.537052 22966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:30.538249 22966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:30.550125 22990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:30.550144 22991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.550310 22966 server_base.cc:1034] running on GCE node
W20260928 14:46:30.550321 22993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.550979 22966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:30.551829 22966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:30.553033 22966 hybrid_clock.cc:648] HybridClock initialized: now 1790606790553007 us; error 54 us; skew 500 ppm
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606790, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:30.556267 22966 init.cc:377] Logged in from keytab as oryx/127.13.223.65@KRBTEST.COM (short username oryx)
I20260928 14:46:30.557372 22966 webserver.cc:460] Webserver started at http://127.13.223.65:41421/ using document root <none> and password file <none>
I20260928 14:46:30.558069 22966 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:30.558177 22966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:30.562793 22966 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:30.565299 23000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.566826 22966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:30.566977 22966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "0dace2533fd54bb9afeb8674a6193bbd"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "e70bd38713f5d035ffd9a1b1c1f0ea4e"
server_key_iv: "bdd61e26b1f5d8c6063ce4c035e7c5fa"
server_key_version: "encryptionkey@0"
I20260928 14:46:30.567593 22966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:30.581549 22966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:30.585359 22966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:30.585604 22966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:30.586293 22966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:30.587299 22966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:30.587349 22966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.587435 22966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:30.587466 22966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.600468 22966 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:43791
I20260928 14:46:30.600530 23113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:43791 every 8 connection(s)
I20260928 14:46:30.601702 22966 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:30.609622 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 22966
I20260928 14:46:30.610256 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:39155
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:43813
--webserver_port=36623
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:30.617617 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.604071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:41805 (local address 127.13.223.126:43813)
0928 14:46:30.604644 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.604665 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.604953 (+ 288us) server_negotiation.cc:407] Connection header received
0928 14:46:30.606702 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.606715 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.607056 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.607412 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.608704 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.609864 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.611329 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.611900 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.612103 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.616739 (+ 4636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:41805: BlockingRecv error: recv got EOF from 127.13.223.65:41805 (error 108)
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":149,"threads_started":1}
I20260928 14:46:30.617808 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.604252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:41805)
0928 14:46:30.604784 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.604829 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.606150 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.607491 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.607499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.607994 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.608508 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.608524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.610037 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.610040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.611168 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.611186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.612109 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.612114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.612324 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.613026 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.613044 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.616451 (+ 3407us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":146,"threads_started":1}
W20260928 14:46:30.618278 23114 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.126:43813: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:30.624830 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.618980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:45759 (local address 127.13.223.126:43813)
0928 14:46:30.619144 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.619151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.619169 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:30.619215 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.619219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.619272 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.619362 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.619819 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.620387 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.621234 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.621496 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.621612 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.624696 (+ 3084us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:45759: BlockingRecv error: recv got EOF from 127.13.223.65:45759 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:46:30.624838 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.618884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:45759)
0928 14:46:30.618990 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.619006 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.619111 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.619389 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.619392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.619575 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.619683 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.619688 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.620508 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.620513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.621101 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.621110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.621617 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.621620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.621719 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.622273 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.622286 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.624525 (+ 2239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:30.631847 23117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.625644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:45543)
0928 14:46:30.625848 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.625863 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.625988 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.626281 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.626285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.626477 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.626638 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.626645 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.627532 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.627538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.628148 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.628157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.628676 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.628680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.628772 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.629355 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.629370 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.631695 (+ 2325us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 14:46:30.632082 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.625850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:45543 (local address 127.13.223.126:43813)
0928 14:46:30.625997 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.626002 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.626020 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:30.626080 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.626083 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.626144 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.626247 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.626766 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.627394 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.628292 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.628545 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.628670 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.631957 (+ 3287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:45543: BlockingRecv error: recv got EOF from 127.13.223.65:45543 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 14:46:30.632143 23114 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:46:30.732987 23118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:30.733357 23118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:30.733439 23118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:30.738023 23118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:30.738128 23118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:30.738229 23118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:30.744004 23118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.66.keytab
--principal=oryx/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:39155
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=36623
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:30.745393 23118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:30.746474 23118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:30.757189 23124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.757225 23118 server_base.cc:1034] running on GCE node
W20260928 14:46:30.757180 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:30.757180 23123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.757956 23118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:30.758734 23118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:30.759939 23118 hybrid_clock.cc:648] HybridClock initialized: now 1790606790759918 us; error 33 us; skew 500 ppm
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606790, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:30.762883 23118 init.cc:377] Logged in from keytab as oryx/127.13.223.66@KRBTEST.COM (short username oryx)
I20260928 14:46:30.763773 23118 webserver.cc:460] Webserver started at http://127.13.223.66:36623/ using document root <none> and password file <none>
I20260928 14:46:30.764439 23118 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:30.764519 23118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:30.768530 23118 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:30.770876 23133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.772424 23118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:30.772533 23118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "e21a862895fc41269db7c88f61955b1c"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "ca236b7acf393b9594486393e8ba7e0a"
server_key_iv: "80b7e56c09bc2b9aee43cfd49bdc52f0"
server_key_version: "encryptionkey@0"
I20260928 14:46:30.773138 23118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:30.801404 23118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:30.804877 23118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:30.805119 23118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:30.805760 23118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:30.806684 23118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:30.806758 23118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.806835 23118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:30.806885 23118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.816416 23118 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:39155
I20260928 14:46:30.816438 23246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:39155 every 8 connection(s)
I20260928 14:46:30.817631 23118 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:30.818885 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23118
I20260928 14:46:30.819407 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:36929
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:43813
--webserver_port=43523
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:30.831048 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.820028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:54979 (local address 127.13.223.126:43813)
0928 14:46:30.820208 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.820216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.821123 (+ 907us) server_negotiation.cc:407] Connection header received
0928 14:46:30.822584 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.822591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.822668 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.822784 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.824059 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.825042 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.826533 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.827003 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.827143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.830905 (+ 3762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:54979: BlockingRecv error: recv got EOF from 127.13.223.66:54979 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 14:46:30.831668 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.820418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:54979)
0928 14:46:30.820946 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.820999 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.822300 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.822962 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.822970 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.823339 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.823885 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.823900 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.825225 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.825229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.826387 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.826399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.827205 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.827208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.827380 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.828142 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.828161 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.830710 (+ 2549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":155,"threads_started":1}
W20260928 14:46:30.832230 23247 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.126:43813: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:30.838315 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.832715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:34829)
0928 14:46:30.832833 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.832848 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.832992 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.833278 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.833281 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.833462 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.833566 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.833572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.834365 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.834369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.835068 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.835080 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.835556 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.835560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.835658 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.836260 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.836272 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.838146 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:46:30.838426 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.832823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:34829 (local address 127.13.223.126:43813)
0928 14:46:30.832939 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.832944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.832960 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:30.833129 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.833132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.833175 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.833260 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.833691 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.834245 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.835194 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.835435 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.835551 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.838291 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:34829: BlockingRecv error: recv got EOF from 127.13.223.66:34829 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:30.845559 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:30.839296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:53853)
0928 14:46:30.839442 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:30.839458 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:30.839546 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:30.839840 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:30.839843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:30.840027 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:30.840133 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.840140 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.841083 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.841086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.841937 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:30.841950 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.842435 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.842438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:30.842521 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.843137 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:30.843153 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:30.845415 (+ 2262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 14:46:30.845868 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:30.839366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:53853 (local address 127.13.223.126:43813)
0928 14:46:30.839499 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:30.839503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:30.839519 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:30.839663 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:30.839665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:30.839713 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:30.839805 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:30.840321 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.840953 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.842085 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:30.842315 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:30.842411 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:30.845650 (+ 3239us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:53853: BlockingRecv error: recv got EOF from 127.13.223.66:53853 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 14:46:30.846005 23247 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:46:30.942032 23249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:30.942361 23249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:30.942409 23249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:30.947528 23249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:30.947610 23249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:30.947675 23249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:30.953533 23249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/krb5kdc/oryx_127.13.223.67.keytab
--principal=oryx/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:36929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=43523
--tserver_master_addrs=127.13.223.126:43813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:30.954646 23249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:30.955664 23249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:30.966727 23258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:30.966747 23256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:30.966747 23255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:30.966809 23249 server_base.cc:1034] running on GCE node
I20260928 14:46:30.967620 23249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:30.968415 23249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:30.969646 23249 hybrid_clock.cc:648] HybridClock initialized: now 1790606790969589 us; error 76 us; skew 500 ppm
Sep 28 14:46:30 dist-test-slave-gkhx krb5kdc[22383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606790, etypes {rep=17 tkt=17 ses=17}, oryx/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:30.973311 23249 init.cc:377] Logged in from keytab as oryx/127.13.223.67@KRBTEST.COM (short username oryx)
I20260928 14:46:30.974359 23249 webserver.cc:460] Webserver started at http://127.13.223.67:43523/ using document root <none> and password file <none>
I20260928 14:46:30.975081 23249 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:30.975176 23249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:30.980034 23249 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:30.982735 23265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:30.984395 23249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:30.984575 23249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "b9fb64f6063a42a48ed930225c68f9f9"
format_stamp: "Formatted at 2026-09-28 14:46:29 on dist-test-slave-gkhx"
server_key: "a8ebf17d25297b50d272b907e0db353d"
server_key_iv: "edbf83c4b65cdf91665faae78545be74"
server_key_version: "encryptionkey@0"
I20260928 14:46:30.985236 23249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:31.003491 23249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:31.007440 23249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:31.007681 23249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:31.008437 23249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:31.009572 23249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:31.009620 23249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:31.009673 23249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:31.009694 23249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:31.021036 23249 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:36929
I20260928 14:46:31.021135 23378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:36929 every 8 connection(s)
I20260928 14:46:31.022289 23249 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:31.027285 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23249
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
W20260928 14:46:31.036778 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:31.024783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:49125 (local address 127.13.223.126:43813)
0928 14:46:31.025030 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.025034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.026062 (+ 1028us) server_negotiation.cc:407] Connection header received
0928 14:46:31.027826 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.027830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.027894 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.027985 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.029521 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.030259 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.031811 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.032064 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.032196 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.036648 (+ 4452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:49125: BlockingRecv error: recv got EOF from 127.13.223.67:49125 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:46:31.037672 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.025196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:49125)
0928 14:46:31.025774 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:31.025908 (+ 134us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:31.027568 (+ 1660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:31.028226 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:31.028237 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:31.028797 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:31.029357 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.029373 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.030415 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.030419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.031671 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.031682 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.032247 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.032251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.032454 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.033218 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:31.033245 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:31.036496 (+ 3251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":150,"threads_started":1}
W20260928 14:46:31.038206 23379 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:43813 (0 consecutive failures): Not authorized: Failed to ping master at 127.13.223.126:43813: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
I20260928 14:46:31.039662 23383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.028411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34840 (local address 127.13.223.126:43813)
0928 14:46:31.029656 (+ 1245us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.029660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.029673 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:46:31.029707 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.029710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.029754 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.029867 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.030857 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.031547 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.032453 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.032689 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.032798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.034375 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:31.034384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:31.034416 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:31.034451 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:31.037029 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:31.037509 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:31.037516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:31.037523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:31.037687 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:31.038080 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:31.038083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:31.038085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:31.038614 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:31.038867 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:31.039225 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 14:46:31.039516 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1165,"thread_start_us":90,"threads_started":1}
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:31.045203 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.038751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:47801)
0928 14:46:31.038988 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:31.039010 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:31.039085 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:31.039346 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:31.039349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:31.039514 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:31.039613 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.039618 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.040444 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.040449 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.041267 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.041279 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.042093 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.042103 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.042192 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.042858 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:31.042871 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:31.045086 (+ 2215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169}
W20260928 14:46:31.045466 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:31.038876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:47801 (local address 127.13.223.126:43813)
0928 14:46:31.038988 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.038993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.039051 (+ 58us) server_negotiation.cc:407] Connection header received
0928 14:46:31.039184 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.039188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.039237 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.039319 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.039737 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.040328 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.041481 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.041796 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.041907 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.045282 (+ 3375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:47801: BlockingRecv error: recv got EOF from 127.13.223.67:47801 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:31.051564 23381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.045836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:42541)
0928 14:46:31.045983 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:31.045996 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:31.046091 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:31.046397 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:31.046400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:31.046618 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:31.046705 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.046711 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.047512 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.047516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.048068 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.048076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.048551 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.048554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.048637 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.049180 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:31.049201 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:31.051396 (+ 2195us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260928 14:46:31.051671 23116 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:31.045994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:42541 (local address 127.13.223.126:43813)
0928 14:46:31.046195 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.046199 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.046211 (+ 12us) server_negotiation.cc:407] Connection header received
0928 14:46:31.046256 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.046257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.046293 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.046381 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.046819 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.047344 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.048191 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.048437 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.048548 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.051549 (+ 3001us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:42541: BlockingRecv error: recv got EOF from 127.13.223.67:42541 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260928 14:46:31.051860 23379 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:31.639745 23384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.633107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:50495)
0928 14:46:31.633440 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:31.633456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:31.633570 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:31.633896 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:31.633900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:31.634100 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:31.634238 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.634244 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.635144 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.635147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.635849 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.635861 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.636467 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.636470 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.636616 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.637342 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:31.637363 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:31.639621 (+ 2258us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":99,"threads_started":1}
W20260928 14:46:31.640089 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:31.633203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:50495 (local address 127.13.223.126:43813)
0928 14:46:31.633480 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.633484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.633520 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:46:31.633677 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.633680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.633732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.633818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.634360 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.635005 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.636007 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.636297 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.636412 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.639943 (+ 3531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:50495: BlockingRecv error: recv got EOF from 127.13.223.65:50495 (error 108)
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":80,"threads_started":1}
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:31 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:31.853097 23386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:31.846998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:48281)
0928 14:46:31.847314 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:31.847329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:31.847491 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:31.847846 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:31.847850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:31.848108 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:31.848220 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.848227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.849144 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.849147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.849827 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:31.849837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.850343 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.850347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:31.850425 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.850971 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:31.850984 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:31.852959 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":94,"threads_started":1}
W20260928 14:46:31.853503 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:31.847164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:48281 (local address 127.13.223.126:43813)
0928 14:46:31.847312 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:31.847317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:31.847447 (+ 130us) server_negotiation.cc:407] Connection header received
0928 14:46:31.847631 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:31.847637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:31.847700 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:31.847789 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:31.848339 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.849033 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.849982 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:31.850203 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:31.850283 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:31.853246 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:48281: BlockingRecv error: recv got EOF from 127.13.223.66:48281 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:32.060300 23387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:32.052826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:38079)
0928 14:46:32.053147 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:32.053162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:32.053267 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:32.053667 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:32.053670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:32.054100 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:32.054252 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.054260 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.055335 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.055339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.056025 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.056038 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.056580 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.056584 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.056689 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.057516 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:32.057531 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:32.060117 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":125,"threads_started":1}
W20260928 14:46:32.060367 23385 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:32.053021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:38079 (local address 127.13.223.126:43813)
0928 14:46:32.053182 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:32.053186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:32.053216 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:46:32.053438 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:32.053441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:32.053493 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:32.053587 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:32.054501 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.055147 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.056172 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.056411 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.056504 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.060257 (+ 3753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:38079: BlockingRecv error: recv got EOF from 127.13.223.67:38079 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:32.647217 23388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:32.640728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:40897)
0928 14:46:32.641068 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:32.641088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:32.641208 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:32.641572 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:32.641575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:32.641843 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:32.641967 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.641974 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.643045 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.643048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.643679 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.643693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.644208 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.644211 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.644298 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.644908 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:32.644931 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:32.647064 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":146,"threads_started":1}
W20260928 14:46:32.647480 23389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:32.640997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:40897 (local address 127.13.223.126:43813)
0928 14:46:32.641226 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:32.641230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:32.641273 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:46:32.641328 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:32.641331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:32.641391 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:32.641526 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:32.642113 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.642930 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.643806 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.644052 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.644141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.647302 (+ 3161us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:40897: BlockingRecv error: recv got EOF from 127.13.223.65:40897 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":70,"threads_started":1}
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:32 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:32.860373 23390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:32.854016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:32897)
0928 14:46:32.854292 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:32.854305 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:32.854407 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:32.854891 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:32.854895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:32.855152 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:32.855261 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.855268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.856227 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.856230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.856800 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:32.856808 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.857436 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.857439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:32.857514 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.858166 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:32.858179 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:32.860211 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":95,"threads_started":1}
W20260928 14:46:32.860598 23389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:32.854082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:32897 (local address 127.13.223.126:43813)
0928 14:46:32.854268 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:32.854272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:32.854381 (+ 109us) server_negotiation.cc:407] Connection header received
0928 14:46:32.854601 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:32.854604 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:32.854665 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:32.854823 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:32.855404 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.856090 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.856949 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:32.857234 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:32.857404 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:32.860459 (+ 3055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:32897: BlockingRecv error: recv got EOF from 127.13.223.66:32897 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:33.068138 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:33.061337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:41953)
0928 14:46:33.061721 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:33.061741 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:33.061856 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:33.062271 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:33.062274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:33.062495 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:33.062688 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.062699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.063717 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.063721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.064419 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.064430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.064934 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.064937 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.065041 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.065596 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:33.065609 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:33.067965 (+ 2356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":104,"threads_started":1}
W20260928 14:46:33.068396 23389 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:33.061606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41953 (local address 127.13.223.126:43813)
0928 14:46:33.061778 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:33.061782 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:33.061811 (+ 29us) server_negotiation.cc:407] Connection header received
0928 14:46:33.061987 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:33.061993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:33.062079 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:33.062166 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:33.062821 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.063559 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.064565 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.064790 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.064875 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.068177 (+ 3302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:41953: BlockingRecv error: recv got EOF from 127.13.223.67:41953 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:33.655421 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:33.648308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:39851)
0928 14:46:33.648681 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:33.648698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:33.648816 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:33.649235 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:33.649238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:33.649450 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:33.649563 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.649570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.650518 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.650521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.651307 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.651318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.651914 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.651917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.652001 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.652637 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:33.652650 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:33.655209 (+ 2559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":107,"threads_started":1}
W20260928 14:46:33.655510 23393 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:33.648527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:39851 (local address 127.13.223.126:43813)
0928 14:46:33.648765 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:33.648769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:33.648787 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:33.648979 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:33.648982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:33.649046 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:33.649145 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:33.649696 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.650390 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.651469 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.651756 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.651867 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.655367 (+ 3500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:39851: BlockingRecv error: recv got EOF from 127.13.223.65:39851 (error 108)
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":69,"threads_started":1}
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:33 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:33.868058 23394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:33.861268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:55845)
0928 14:46:33.861678 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:33.861696 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:33.861884 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:33.862212 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:33.862215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:33.862456 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:33.862596 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.862604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.863485 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.863491 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.864140 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:33.864150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.864674 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.864678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:33.864777 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.865416 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:33.865429 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:33.867867 (+ 2438us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":158,"threads_started":1}
W20260928 14:46:33.868119 23393 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:33.861467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:55845 (local address 127.13.223.126:43813)
0928 14:46:33.861673 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:33.861678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:33.861801 (+ 123us) server_negotiation.cc:407] Connection header received
0928 14:46:33.862005 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:33.862011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:33.862074 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:33.862189 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:33.862735 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.863351 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.864278 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:33.864541 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:33.864668 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:33.868002 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:55845: BlockingRecv error: recv got EOF from 127.13.223.66:55845 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:34.075229 23395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:34.069034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:41635)
0928 14:46:34.069317 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:34.069331 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:34.069438 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:34.069678 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:34.069680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:34.069879 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:34.070093 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.070107 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.070969 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.070975 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.071631 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.071640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.072184 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.072187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.072267 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.072996 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:34.073010 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:34.075086 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":109,"threads_started":1}
W20260928 14:46:34.075464 23393 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:34.069146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41635 (local address 127.13.223.126:43813)
0928 14:46:34.069283 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:34.069288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:34.069401 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:46:34.069522 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:34.069525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:34.069568 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:34.069638 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:34.070223 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.070833 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.071759 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.072057 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.072177 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.075304 (+ 3127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:41635: BlockingRecv error: recv got EOF from 127.13.223.67:41635 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:34.668623 23396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:34.656464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:41247)
0928 14:46:34.656805 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:34.656821 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:34.656964 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:34.657262 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:34.657265 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:34.657466 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:34.657571 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.657577 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.658399 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.658402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.659112 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.659122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.659669 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.659673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.659775 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.666160 (+ 6385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:34.666177 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:34.668492 (+ 2315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
W20260928 14:46:34.668956 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:34.656635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:41247 (local address 127.13.223.126:43813)
0928 14:46:34.656936 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:34.656940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:34.656958 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:46:34.657059 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:34.657062 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:34.657104 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:34.657188 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:34.657692 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.658275 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.659277 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.659535 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.659626 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.668700 (+ 9074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:41247: BlockingRecv error: recv got EOF from 127.13.223.65:41247 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":63,"threads_started":1}
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:34 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:34.876401 23398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:34.869162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:60909)
0928 14:46:34.869808 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:34.869824 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:34.869998 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:34.870423 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:34.870427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:34.870716 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:34.870912 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.870922 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.872021 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.872026 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.872867 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:34.872879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.873341 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.873344 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:34.873419 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.873978 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:34.873991 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:34.876250 (+ 2259us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":283,"threads_started":1}
W20260928 14:46:34.876668 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:34.869432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:60909 (local address 127.13.223.126:43813)
0928 14:46:34.869679 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:34.869685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:34.869975 (+ 290us) server_negotiation.cc:407] Connection header received
0928 14:46:34.870174 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:34.870178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:34.870235 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:34.870335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:34.871071 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.871868 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.872989 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:34.873212 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:34.873296 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:34.876532 (+ 3236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:60909: BlockingRecv error: recv got EOF from 127.13.223.66:60909 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:35.083469 23399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:35.076364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:57115)
0928 14:46:35.076723 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:35.076738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:35.077018 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:35.077482 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:35.077484 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:35.077666 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:35.077778 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.077783 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.078912 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.078916 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.079562 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.079572 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.080263 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.080268 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.080394 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.081012 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:35.081024 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:35.083336 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":111,"threads_started":1}
W20260928 14:46:35.083616 23397 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:35.076420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:57115 (local address 127.13.223.126:43813)
0928 14:46:35.076658 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:35.076664 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:35.076900 (+ 236us) server_negotiation.cc:407] Connection header received
0928 14:46:35.077252 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:35.077256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:35.077320 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:35.077413 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:35.077932 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.078674 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.079721 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.080128 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.080226 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.083486 (+ 3260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:57115: BlockingRecv error: recv got EOF from 127.13.223.67:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:35.676412 23400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:35.669814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:43937)
0928 14:46:35.670116 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:35.670132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:35.670229 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:35.670652 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:35.670654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:35.670989 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:35.671124 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.671133 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.672113 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.672117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.672780 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.672790 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.673334 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.673337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.673423 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.674063 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:35.674082 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:35.676268 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":101,"threads_started":1}
W20260928 14:46:35.676651 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:35.669939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:43937 (local address 127.13.223.126:43813)
0928 14:46:35.670291 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:35.670298 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:35.670318 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:46:35.670390 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:35.670393 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:35.670443 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:35.670612 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:35.671275 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.671980 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.672958 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.673200 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.673291 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.676474 (+ 3183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:43937: BlockingRecv error: recv got EOF from 127.13.223.65:43937 (error 108)
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":74,"threads_started":1}
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:35 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:35.883805 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:35.877401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:36695)
0928 14:46:35.877815 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:35.877833 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:35.877959 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:35.878257 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:35.878260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:35.878485 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:35.878618 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.878624 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.879464 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.879469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.880060 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:35.880068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.880563 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.880567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:35.880658 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.881250 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:35.881266 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:35.883610 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":167,"threads_started":1}
W20260928 14:46:35.884138 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:35.877618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:36695 (local address 127.13.223.126:43813)
0928 14:46:35.877817 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:35.877824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:35.877956 (+ 132us) server_negotiation.cc:407] Connection header received
0928 14:46:35.878080 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:35.878087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:35.878133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:35.878228 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:35.878770 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.879339 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.880192 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:35.880435 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:35.880552 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:35.883815 (+ 3263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:36695: BlockingRecv error: recv got EOF from 127.13.223.66:36695 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:36.091591 23403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:36.084455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:47157)
0928 14:46:36.084845 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:36.084948 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:36.085087 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:36.085521 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:36.085525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:36.085778 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:36.085891 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.085899 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.086780 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.086783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.087524 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.087535 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.088105 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.088107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.088205 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.088795 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:36.088807 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:36.091459 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":104,"threads_started":1}
W20260928 14:46:36.091827 23401 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:36.084559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:47157 (local address 127.13.223.126:43813)
0928 14:46:36.084737 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:36.084741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:36.085039 (+ 298us) server_negotiation.cc:407] Connection header received
0928 14:46:36.085243 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:36.085246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:36.085347 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:36.085466 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:36.086012 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.086642 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.087666 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.087965 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.088079 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.091658 (+ 3579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:47157: BlockingRecv error: recv got EOF from 127.13.223.67:47157 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:36.684592 23404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:36.677636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:35365)
0928 14:46:36.678044 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:36.678101 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:36.678237 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:36.678621 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:36.678624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:36.678837 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:36.678947 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.678952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.679921 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.679924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.680661 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.680672 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.681214 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.681217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.681291 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.681824 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:36.681837 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:36.684464 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
W20260928 14:46:36.684935 23405 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:36.677825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:35365 (local address 127.13.223.126:43813)
0928 14:46:36.678075 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:36.678079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:36.678200 (+ 121us) server_negotiation.cc:407] Connection header received
0928 14:46:36.678388 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:36.678390 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:36.678442 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:36.678529 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:36.679123 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.679789 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.680797 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.681072 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.681169 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.684713 (+ 3544us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:35365: BlockingRecv error: recv got EOF from 127.13.223.65:35365 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:36 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:36.891657 23406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:36.884856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:47213)
0928 14:46:36.885220 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:36.885234 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:36.885394 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:36.885704 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:36.885708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:36.886023 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:36.886129 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.886136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.887089 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.887092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.887715 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:36.887723 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.888258 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.888263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:36.888357 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.889201 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:36.889220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:36.891532 (+ 2312us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":94,"threads_started":1}
W20260928 14:46:36.891871 23405 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:36.884858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:47213 (local address 127.13.223.126:43813)
0928 14:46:36.885124 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:36.885129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:36.885326 (+ 197us) server_negotiation.cc:407] Connection header received
0928 14:46:36.885510 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:36.885514 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:36.885575 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:36.885677 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:36.886273 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.886939 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.887858 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:36.888112 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:36.888253 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:36.891738 (+ 3485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:47213: BlockingRecv error: recv got EOF from 127.13.223.66:47213 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:37.099162 23407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:37.092538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:39177)
0928 14:46:37.092866 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:37.092884 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:37.093034 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:37.093337 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:37.093344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:37.093611 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:37.093778 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.093786 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.094700 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.094703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.095438 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.095448 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.096021 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.096024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.096096 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.096726 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:37.096745 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:37.099025 (+ 2280us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":110,"threads_started":1}
W20260928 14:46:37.099337 23405 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:37.092690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:39177 (local address 127.13.223.126:43813)
0928 14:46:37.092853 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:37.092857 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:37.093012 (+ 155us) server_negotiation.cc:407] Connection header received
0928 14:46:37.093125 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:37.093130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:37.093201 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:37.093272 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:37.093923 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.094525 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.095572 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.095898 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.095992 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.099197 (+ 3205us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:39177: BlockingRecv error: recv got EOF from 127.13.223.67:39177 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:37.692375 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:37.685664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:36527)
0928 14:46:37.686206 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:37.686221 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:37.686347 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:37.686645 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:37.686647 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:37.686867 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:37.686977 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.686983 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.687800 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.687804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.688521 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.688532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.689257 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.689261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.689337 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.690032 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:37.690046 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:37.692239 (+ 2193us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":120,"threads_started":1}
W20260928 14:46:37.692629 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:37.685845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:36527 (local address 127.13.223.126:43813)
0928 14:46:37.686174 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:37.686178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:37.686326 (+ 148us) server_negotiation.cc:407] Connection header received
0928 14:46:37.686442 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:37.686445 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:37.686493 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:37.686621 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:37.687107 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.687678 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.688691 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.689056 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.689186 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.692475 (+ 3289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:36527: BlockingRecv error: recv got EOF from 127.13.223.65:36527 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":91,"threads_started":1}
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:37 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:37.898427 23410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:37.892499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:33547)
0928 14:46:37.892818 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:37.892836 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:37.892948 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:37.893249 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:37.893253 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:37.893460 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:37.893562 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.893568 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.894362 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.894365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.894976 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:37.894985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.895431 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.895434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:37.895501 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.896061 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:37.896073 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:37.898156 (+ 2083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":107,"threads_started":1}
W20260928 14:46:37.898591 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:37.892625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:33547 (local address 127.13.223.126:43813)
0928 14:46:37.892823 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:37.892827 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:37.892918 (+ 91us) server_negotiation.cc:407] Connection header received
0928 14:46:37.893035 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:37.893038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:37.893083 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:37.893159 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:37.893679 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.894248 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.895092 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:37.895323 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:37.895404 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:37.898412 (+ 3008us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:33547: BlockingRecv error: recv got EOF from 127.13.223.66:33547 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:38.106387 23411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:38.100220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:52769)
0928 14:46:38.100616 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:38.100629 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:38.100734 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:38.101048 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:38.101051 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:38.101252 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:38.101361 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.101367 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.102318 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.102322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.102983 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.102994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.103551 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.103554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.103628 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.104283 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:38.104297 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:38.106255 (+ 1958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":100,"threads_started":1}
W20260928 14:46:38.106673 23409 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:38.100376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:52769 (local address 127.13.223.126:43813)
0928 14:46:38.100600 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:38.100604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:38.100707 (+ 103us) server_negotiation.cc:407] Connection header received
0928 14:46:38.100830 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:38.100833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:38.100883 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:38.100972 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:38.101508 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.102172 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.103134 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.103414 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.103522 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.106506 (+ 2984us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:52769: BlockingRecv error: recv got EOF from 127.13.223.67:52769 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:38.700268 23412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:38.693760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:55229)
0928 14:46:38.694085 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:38.694099 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:38.694236 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:38.694754 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:38.694758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:38.694994 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:38.695103 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.695135 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.696195 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.696199 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.696800 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.696809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.697320 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.697324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.697419 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.698093 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:38.698106 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:38.700083 (+ 1977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":90,"threads_started":1}
W20260928 14:46:38.700371 23413 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:38.694057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:55229 (local address 127.13.223.126:43813)
0928 14:46:38.694400 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:38.694405 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:38.694453 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:46:38.694494 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:38.694498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:38.694582 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:38.694693 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:38.695320 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.696071 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.696934 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.697189 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.697317 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.700225 (+ 2908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:55229: BlockingRecv error: recv got EOF from 127.13.223.65:55229 (error 108)
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:38 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:38.906250 23414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:38.900068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:57647)
0928 14:46:38.900463 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:38.900480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:38.900604 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:38.900902 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:38.900905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:38.901166 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:38.901302 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.901309 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.902288 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.902294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.902924 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:38.902933 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.903428 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.903432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:38.903519 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.904116 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:38.904129 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:38.906118 (+ 1989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":80,"threads_started":1}
W20260928 14:46:38.906591 23413 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:38.900206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:57647 (local address 127.13.223.126:43813)
0928 14:46:38.900361 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:38.900367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:38.900572 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:46:38.900717 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:38.900720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:38.900776 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:38.900869 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:38.901441 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.902172 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.903051 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:38.903292 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:38.903423 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:38.906444 (+ 3021us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:57647: BlockingRecv error: recv got EOF from 127.13.223.66:57647 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:39.114408 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:39.107521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:55771)
0928 14:46:39.107858 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:39.107877 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:39.108004 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:39.108306 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:39.108308 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:39.108543 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:39.108655 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.108661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.109634 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.109637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.110243 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.110253 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.110943 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.110946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.111032 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.111801 (+ 769us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:39.111819 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:39.114248 (+ 2429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":135,"threads_started":1}
W20260928 14:46:39.114679 23413 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:39.107769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:55771 (local address 127.13.223.126:43813)
0928 14:46:39.107968 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:39.107972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:39.107994 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:46:39.108108 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:39.108111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:39.108163 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:39.108249 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:39.108882 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.109436 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.110402 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.110697 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.110897 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.114500 (+ 3603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:55771: BlockingRecv error: recv got EOF from 127.13.223.67:55771 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:39.713547 23416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:39.701386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:59291)
0928 14:46:39.701714 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:39.701728 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:39.701830 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:39.702117 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:39.702119 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:39.702332 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:39.702432 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.702438 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.703272 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.703275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.703866 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.703874 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.704368 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.704371 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.704448 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.711144 (+ 6696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:39.711162 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:39.713393 (+ 2231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":95,"threads_started":1}
W20260928 14:46:39.713966 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:39.701552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:59291 (local address 127.13.223.126:43813)
0928 14:46:39.701795 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:39.701799 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:39.701815 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:39.701944 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:39.701952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:39.701997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:39.702078 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:39.702590 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.703149 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.703977 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.704205 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.704287 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.713816 (+ 9529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:59291: BlockingRecv error: recv got EOF from 127.13.223.65:59291 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":60,"threads_started":1}
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:39 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:39.913676 23418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:39.907327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:35595)
0928 14:46:39.907619 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:39.907633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:39.907796 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:39.908078 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:39.908080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:39.908275 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:39.908375 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.908380 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.909296 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.909299 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.909922 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:39.909932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.910527 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.910530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:39.910695 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.911499 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:39.911521 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:39.913534 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
W20260928 14:46:39.914132 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:39.907476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:35595 (local address 127.13.223.126:43813)
0928 14:46:39.907628 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:39.907633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:39.907702 (+ 69us) server_negotiation.cc:407] Connection header received
0928 14:46:39.907895 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:39.907898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:39.907954 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:39.908041 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:39.908521 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.909135 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.910075 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:39.910393 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:39.910492 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:39.913792 (+ 3300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:35595: BlockingRecv error: recv got EOF from 127.13.223.66:35595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:40.122237 23419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:40.115391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:38127)
0928 14:46:40.115707 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:40.115721 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:40.115900 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:40.116164 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:40.116166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:40.116354 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:40.116464 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.116470 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.117476 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.117479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.118244 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.118256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.118982 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.118986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.119064 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.119701 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:40.119718 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:40.122084 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":123,"threads_started":1}
W20260928 14:46:40.122525 23417 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:40.115477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:38127 (local address 127.13.223.126:43813)
0928 14:46:40.115635 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:40.115640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:40.115796 (+ 156us) server_negotiation.cc:407] Connection header received
0928 14:46:40.115994 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:40.115997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:40.116045 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:40.116118 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:40.116643 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.117311 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.118475 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.118761 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.118991 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.122310 (+ 3319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:38127: BlockingRecv error: recv got EOF from 127.13.223.67:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:40.721338 23420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:40.714523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:52135)
0928 14:46:40.715011 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:40.715028 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:40.715152 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:40.715512 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:40.715516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:40.715757 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:40.715943 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.715954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.716859 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.716862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.717576 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.717590 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.718310 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.718313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.718416 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.719042 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:40.719055 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:40.721152 (+ 2097us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":83,"threads_started":1}
W20260928 14:46:40.721473 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:40.714620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:52135 (local address 127.13.223.126:43813)
0928 14:46:40.714966 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:40.714972 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:40.715142 (+ 170us) server_negotiation.cc:407] Connection header received
0928 14:46:40.715257 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:40.715260 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:40.715324 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:40.715433 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:40.716099 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.716719 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.717761 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.718141 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.718292 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.721309 (+ 3017us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:52135: BlockingRecv error: recv got EOF from 127.13.223.65:52135 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":79,"threads_started":1}
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:40 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:40.920549 23422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:40.914654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:37375)
0928 14:46:40.914967 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:40.914981 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:40.915080 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:40.915319 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:40.915322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:40.915503 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:40.915602 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.915607 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.916392 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.916395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.917042 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:40.917051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.917505 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.917508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:40.917583 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.918160 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:40.918172 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:40.920423 (+ 2251us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
W20260928 14:46:40.920748 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:40.914687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:37375 (local address 127.13.223.126:43813)
0928 14:46:40.914821 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:40.914824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:40.915051 (+ 227us) server_negotiation.cc:407] Connection header received
0928 14:46:40.915165 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:40.915167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:40.915211 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:40.915278 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:40.915699 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.916275 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.917187 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:40.917411 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:40.917483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:40.920603 (+ 3120us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:37375: BlockingRecv error: recv got EOF from 127.13.223.66:37375 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:41.130261 23423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:41.123554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:32793)
0928 14:46:41.123956 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:41.123974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:41.124133 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:41.124483 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:41.124486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:41.124786 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:41.124891 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.124897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.125833 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.125837 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.126451 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.126461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.127122 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.127125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.127209 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.127882 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:41.127897 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:41.130086 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":111,"threads_started":1}
W20260928 14:46:41.130420 23421 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:41.123768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:32793 (local address 127.13.223.126:43813)
0928 14:46:41.123930 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:41.123938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:41.124109 (+ 171us) server_negotiation.cc:407] Connection header received
0928 14:46:41.124287 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:41.124290 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:41.124340 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:41.124457 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:41.125016 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.125683 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.126646 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.126970 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.127068 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.130252 (+ 3184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:32793: BlockingRecv error: recv got EOF from 127.13.223.67:32793 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:41.729557 23424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:41.722602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:40091)
0928 14:46:41.722891 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:41.722906 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:41.723022 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:41.723608 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:41.723612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:41.723900 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:41.724030 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.724038 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.725113 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.725116 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.725711 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.725722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.726335 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.726338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.726415 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.727037 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:41.727051 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:41.729416 (+ 2365us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
W20260928 14:46:41.729956 23425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:41.722905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:40091 (local address 127.13.223.126:43813)
0928 14:46:41.723199 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:41.723203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:41.723217 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:46:41.723256 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:41.723259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:41.723340 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:41.723496 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:41.724201 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.724944 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.725927 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.726199 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.726289 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.729619 (+ 3330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:40091: BlockingRecv error: recv got EOF from 127.13.223.65:40091 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":77,"threads_started":1}
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:41 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:41.928952 23426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:41.921656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:44731)
0928 14:46:41.922116 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:41.922134 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:41.922298 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:41.922722 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:41.922725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:41.923095 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:41.923255 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.923266 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.924333 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.924336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.924989 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:41.925001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.925588 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.925591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:41.925687 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.926334 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:41.926349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:41.928779 (+ 2430us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":91,"threads_started":1}
W20260928 14:46:41.929280 23425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:41.921885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44731 (local address 127.13.223.126:43813)
0928 14:46:41.922088 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:41.922092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:41.922394 (+ 302us) server_negotiation.cc:407] Connection header received
0928 14:46:41.922454 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:41.922457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:41.922514 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:41.922637 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:41.923431 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.924182 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.925146 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:41.925436 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:41.925535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:41.929145 (+ 3610us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:44731: BlockingRecv error: recv got EOF from 127.13.223.66:44731 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:42.138168 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:42.131225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:57145)
0928 14:46:42.131628 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:42.131647 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:42.131815 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:42.132101 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:42.132104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:42.132328 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:42.132423 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.132429 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.133295 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.133298 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.133872 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.133881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.134432 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.134436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.134525 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.135346 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:42.135363 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:42.138034 (+ 2671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":149,"threads_started":1}
W20260928 14:46:42.138407 23425 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:42.131440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:57145 (local address 127.13.223.126:43813)
0928 14:46:42.131629 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:42.131635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:42.131821 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:46:42.131916 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:42.131919 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:42.131978 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:42.132078 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:42.132552 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.133116 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.134002 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.134306 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.134413 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.138188 (+ 3775us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:57145: BlockingRecv error: recv got EOF from 127.13.223.67:57145 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:42.737686 23428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:42.730795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:48635)
0928 14:46:42.731154 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:42.731169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:42.731276 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:42.731556 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:42.731559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:42.731747 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:42.731850 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.731857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.732993 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.732996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.733793 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.733805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.734311 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.734315 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.734409 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.735133 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:42.735146 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:42.737558 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":78,"threads_started":1}
W20260928 14:46:42.738018 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:42.730839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:48635 (local address 127.13.223.126:43813)
0928 14:46:42.731233 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:42.731237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:42.731254 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:42.731375 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:42.731378 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:42.731424 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:42.731511 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:42.732050 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.732846 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.733956 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.734191 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.734306 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.737772 (+ 3466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:48635: BlockingRecv error: recv got EOF from 127.13.223.65:48635 (error 108)
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":166,"threads_started":1}
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:42 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:42.937284 23430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:42.930067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:51753)
0928 14:46:42.930413 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:42.930427 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:42.930536 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:42.931037 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:42.931040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:42.931252 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:42.931371 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.931378 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.932388 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.932392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.933074 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:42.933087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.933608 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.933611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:42.933699 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.934334 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:42.934347 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:42.937076 (+ 2729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":121,"threads_started":1}
W20260928 14:46:42.937422 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:42.930241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:51753 (local address 127.13.223.126:43813)
0928 14:46:42.930412 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:42.930416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:42.930506 (+ 90us) server_negotiation.cc:407] Connection header received
0928 14:46:42.930787 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:42.930790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:42.930846 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:42.930956 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:42.931527 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.932165 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.933227 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:42.933476 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:42.933598 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:42.937299 (+ 3701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:51753: BlockingRecv error: recv got EOF from 127.13.223.66:51753 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:43.148521 23431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:43.139530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:41755)
0928 14:46:43.139912 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:43.139928 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:43.140047 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:43.140570 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:43.140573 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:43.140796 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:43.140931 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.140939 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.142259 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.142263 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.142999 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.143018 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.144016 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.144021 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.144117 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.144969 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:43.144983 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:43.148380 (+ 3397us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":130,"threads_started":1}
W20260928 14:46:43.148828 23429 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:43.139456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41755 (local address 127.13.223.126:43813)
0928 14:46:43.139614 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:43.139619 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:43.139998 (+ 379us) server_negotiation.cc:407] Connection header received
0928 14:46:43.140203 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:43.140206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:43.140386 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:43.140482 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:43.141149 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.141962 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.143435 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.143708 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.143913 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.148691 (+ 4778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:41755: BlockingRecv error: recv got EOF from 127.13.223.67:41755 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:43.745182 23432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:43.738759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:57125)
0928 14:46:43.739147 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:43.739161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:43.739270 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:43.739580 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:43.739583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:43.739832 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:43.739939 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.739944 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.740854 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.740858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.741511 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.741522 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.742092 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.742096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.742195 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.742907 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:43.742921 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:43.745045 (+ 2124us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":151,"threads_started":1}
W20260928 14:46:43.745404 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:43.738923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:57125 (local address 127.13.223.126:43813)
0928 14:46:43.739213 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:43.739217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:43.739234 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:43.739405 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:43.739408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:43.739457 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:43.739543 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:43.740046 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.740673 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.741659 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.741933 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.742035 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.745224 (+ 3189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:57125: BlockingRecv error: recv got EOF from 127.13.223.65:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":79,"threads_started":1}
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:43 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:43.945187 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:43.938354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:50209)
0928 14:46:43.938709 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:43.938725 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:43.938832 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:43.939217 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:43.939220 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:43.939453 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:43.939551 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.939557 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.940483 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.940487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.941236 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:43.941247 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.941801 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.941805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:43.941969 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.942612 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:43.942633 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:43.945058 (+ 2425us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":123,"threads_started":1}
W20260928 14:46:43.945411 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:43.938686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:50209 (local address 127.13.223.126:43813)
0928 14:46:43.938875 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:43.938878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:43.938895 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:46:43.938939 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:43.938942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:43.938985 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:43.939068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:43.939714 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.940338 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.941411 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:43.941646 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:43.941738 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:43.945210 (+ 3472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:50209: BlockingRecv error: recv got EOF from 127.13.223.66:50209 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:44.155835 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:44.149580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:48051)
0928 14:46:44.149960 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:44.149976 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:44.150094 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:44.150345 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:44.150348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:44.150607 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:44.150730 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.150736 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.151504 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.151506 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.152133 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.152142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.152659 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.152663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.152737 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.153412 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:44.153452 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:44.155641 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":75,"threads_started":1}
W20260928 14:46:44.155880 23433 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:44.149766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:48051 (local address 127.13.223.126:43813)
0928 14:46:44.149944 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:44.149948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:44.150061 (+ 113us) server_negotiation.cc:407] Connection header received
0928 14:46:44.150182 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:44.150184 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:44.150232 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:44.150304 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:44.150842 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.151389 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.152283 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.152517 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.152617 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.155771 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:48051: BlockingRecv error: recv got EOF from 127.13.223.67:48051 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:44.758740 23436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:44.746220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:42285)
0928 14:46:44.746621 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:44.746646 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:44.746772 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:44.747196 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:44.747199 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:44.747450 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:44.747564 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.747569 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.748542 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.748545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.749231 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.749242 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.749810 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.749814 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.749904 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.756353 (+ 6449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:44.756372 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:44.758508 (+ 2136us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":100,"threads_started":1}
W20260928 14:46:44.758982 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:44.746355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:42285 (local address 127.13.223.126:43813)
0928 14:46:44.746718 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:44.746721 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:44.746737 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:44.746930 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:44.746933 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:44.746983 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:44.747171 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:44.747704 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.748406 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.749359 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.749621 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.749714 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.758781 (+ 9067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:42285: BlockingRecv error: recv got EOF from 127.13.223.65:42285 (error 108)
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:44 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:44.952345 23438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:44.946163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:55683)
0928 14:46:44.946482 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:44.946498 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:44.946640 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:44.946958 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:44.946961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:44.947178 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:44.947282 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.947289 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.948077 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.948082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.948679 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:44.948688 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.949171 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.949175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:44.949273 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.949908 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:44.949921 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:44.952163 (+ 2242us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":145,"threads_started":1}
W20260928 14:46:44.952783 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:44.946420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:55683 (local address 127.13.223.126:43813)
0928 14:46:44.946605 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:44.946610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:44.946626 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:46:44.946785 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:44.946793 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:44.946838 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:44.946929 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:44.947412 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.947956 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.948813 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:44.949047 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:44.949159 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:44.952380 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:55683: BlockingRecv error: recv got EOF from 127.13.223.66:55683 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.67@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:45.163858 23439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:45.156807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.67:55341)
0928 14:46:45.157181 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:45.157204 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:45.157402 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:45.157713 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:45.157716 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:45.157960 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:45.158062 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.158069 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.158946 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.158951 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.159560 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.159569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.160074 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.160077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.160167 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.160991 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:45.161007 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:45.163638 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":96,"threads_started":1}
W20260928 14:46:45.164047 23437 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:45.156936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:55341 (local address 127.13.223.126:43813)
0928 14:46:45.157134 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:45.157140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:45.157332 (+ 192us) server_negotiation.cc:407] Connection header received
0928 14:46:45.157517 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:45.157520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:45.157579 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:45.157685 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:45.158196 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.158829 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.159699 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.159944 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.160064 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.163923 (+ 3859us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.67:55341: BlockingRecv error: recv got EOF from 127.13.223.67:55341 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.65@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:45.767256 23440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:45.759991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.65:47957)
0928 14:46:45.760451 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:45.760477 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:45.760612 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:45.761051 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:45.761054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:45.761251 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:45.761357 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.761363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.762447 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.762450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.763107 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.763119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.763605 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.763607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.763741 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.764577 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:45.764590 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:45.767125 (+ 2535us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":98,"threads_started":1}
W20260928 14:46:45.767432 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:45.760170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:47957 (local address 127.13.223.126:43813)
0928 14:46:45.760438 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:45.760442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:45.760601 (+ 159us) server_negotiation.cc:407] Connection header received
0928 14:46:45.760718 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:45.760721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:45.760862 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:45.760952 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:45.761552 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.762291 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.763233 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.763494 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.763582 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.767289 (+ 3707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.65:47957: BlockingRecv error: recv got EOF from 127.13.223.65:47957 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":91,"threads_started":1}
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
Sep 28 14:46:45 dist-test-slave-gkhx krb5kdc[22383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.13.223.66@KRBTEST.COM for oryx/127.13.223.126@KRBTEST.COM, Server not found in Kerberos database
I20260928 14:46:45.960736 23442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:45.953685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43813 (local address 127.13.223.66:48391)
0928 14:46:45.954021 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:45.954039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:45.954155 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:45.954584 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:45.954587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:45.954866 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:45.954984 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.954992 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.955972 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.955978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.956662 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:45.956674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.957392 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.957396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:45.957510 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.958086 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:45.958099 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:45.960511 (+ 2412us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.13.223.126:43813: Server oryx/127.13.223.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
W20260928 14:46:45.960810 23441 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:45.953871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:48391 (local address 127.13.223.126:43813)
0928 14:46:45.954024 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:45.954027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:45.954128 (+ 101us) server_negotiation.cc:407] Connection header received
0928 14:46:45.954318 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:45.954323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:45.954396 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:45.954528 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:45.955131 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.955841 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.956810 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:45.957093 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:45.957382 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:45.960685 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.13.223.66:48391: BlockingRecv error: recv got EOF from 127.13.223.66:48391 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:46:46.029497 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22966
I20260928 14:46:46.035852 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23118
I20260928 14:46:46.042467 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23249
I20260928 14:46:46.049562 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 22896
2026-09-28T14:46:46Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18316 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T14:46:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:46Z Disabled control of system clock
I20260928 14:46:46.084353 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41129
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:41129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 14:46:46.215688 23446 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:46.216056 23446 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:46.216099 23446 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:46.221300 23446 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:46.221426 23446 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:46.221446 23446 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:46.221467 23446 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:46.221483 23446 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:46.227743 23446 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:41129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23446
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:46.229072 23446 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:46.230729 23446 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:46.242803 23452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.242925 23451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:46.242826 23446 server_base.cc:1034] running on GCE node
W20260928 14:46:46.243147 23454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:46.244329 23446 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:46.246011 23446 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:46.247272 23446 hybrid_clock.cc:648] HybridClock initialized: now 1790606806247227 us; error 60 us; skew 500 ppm
I20260928 14:46:46.250232 23446 webserver.cc:460] Webserver started at http://127.13.223.126:35503/ using document root <none> and password file <none>
I20260928 14:46:46.251125 23446 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:46.251255 23446 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:46.251550 23446 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:46.254902 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "b2a3147af7f5410eb51d48c8bc261f3d"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "557f58a711c460146852a4ce0d88487b"
server_key_iv: "71cd171a799d6596cd5908964d3d18a6"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.255759 23446 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "b2a3147af7f5410eb51d48c8bc261f3d"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "557f58a711c460146852a4ce0d88487b"
server_key_iv: "71cd171a799d6596cd5908964d3d18a6"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.261037 23446 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:46:46.264369 23460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.266016 23446 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:46.266249 23446 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "b2a3147af7f5410eb51d48c8bc261f3d"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "557f58a711c460146852a4ce0d88487b"
server_key_iv: "71cd171a799d6596cd5908964d3d18a6"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.266427 23446 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:46.286131 23446 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:46.287117 23446 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:46.287321 23446 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:46.296113 23446 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:41129
I20260928 14:46:46.296136 23512 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:41129 every 8 connection(s)
I20260928 14:46:46.297308 23446 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:46.301391 23513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:46.303242 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23446
I20260928 14:46:46.303390 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:46.303735 14205 external_mini_cluster.cc:1428] Setting key 7f55728d3bee4a3e42788ee427a26251
I20260928 14:46:46.309788 23513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d: Bootstrap starting.
I20260928 14:46:46.312858 23513 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:46.314039 23513 log.cc:824] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:46.315816 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.305742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.13.223.126:41129)
0928 14:46:46.306723 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:46.306819 (+ 96us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:46.306867 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:46:46.307737 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:46.307761 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:46.308255 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:46.308665 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:46.309194 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.310466 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.311476 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.312016 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.312224 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:46.313221 (+ 997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:46.313230 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:46.313269 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:46.313868 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:46.313959 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:46.313989 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:46.314121 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:46.314451 (+ 330us) server_negotiation.cc:299] Negotiation successful
0928 14:46:46.314870 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":174,"threads_started":1}
I20260928 14:46:46.316601 23513 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d: No bootstrap required, opened a new log
I20260928 14:46:46.322013 23513 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } }
I20260928 14:46:46.322443 23513 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:46.322549 23513 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2a3147af7f5410eb51d48c8bc261f3d, State: Initialized, Role: FOLLOWER
I20260928 14:46:46.323228 23513 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } }
I20260928 14:46:46.323488 23513 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:46.323596 23513 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:46.323762 23513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:46.325026 23513 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } }
I20260928 14:46:46.325510 23513 leader_election.cc:302] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2a3147af7f5410eb51d48c8bc261f3d; no voters:
I20260928 14:46:46.325930 23513 leader_election.cc:288] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:46.326135 23518 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:46.326634 23518 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [term 1 LEADER]: Becoming Leader. State: Replica: b2a3147af7f5410eb51d48c8bc261f3d, State: Running, Role: LEADER
I20260928 14:46:46.327114 23518 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } }
I20260928 14:46:46.327960 23513 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:46.329149 23519 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } } }
I20260928 14:46:46.329098 23520 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2a3147af7f5410eb51d48c8bc261f3d. Latest consensus state: current_term: 1 leader_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2a3147af7f5410eb51d48c8bc261f3d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41129 } } }
I20260928 14:46:46.329300 23520 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:46.329559 23519 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:46.330687 23527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:46.334260 23527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:46.340667 23527 catalog_manager.cc:1347] Generated new cluster ID: 595d79799e554dba8e6d6c5c8c52b9fa
I20260928 14:46:46.340766 23527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:46.362903 23527 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:46.364524 23527 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:46.371745 23527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2a3147af7f5410eb51d48c8bc261f3d: Generated new TSK 0
I20260928 14:46:46.373400 23527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:46.380467 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--builtin_ntp_servers=127.13.223.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 14:46:46.511297 23537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:46.511602 23537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:46.511644 23537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:46.516131 23537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:46.516227 23537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:46.516289 23537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:46.521665 23537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:46.523008 23537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:46.524551 23537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:46.537720 23543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.537732 23542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.537781 23545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:46.537985 23537 server_base.cc:1034] running on GCE node
I20260928 14:46:46.538654 23537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:46.539440 23537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:46.540627 23537 hybrid_clock.cc:648] HybridClock initialized: now 1790606806540605 us; error 41 us; skew 500 ppm
I20260928 14:46:46.542814 23537 webserver.cc:460] Webserver started at http://127.13.223.65:38983/ using document root <none> and password file <none>
I20260928 14:46:46.543481 23537 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:46.543556 23537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:46.543787 23537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:46.546103 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "d74ebf24dff1417ba4ed4f6b710f52e6"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "909c18474a65b944f83b6e3f837d776b"
server_key_iv: "cb626ffe3970e917b0463e3f091c7cbc"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.546772 23537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "d74ebf24dff1417ba4ed4f6b710f52e6"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "909c18474a65b944f83b6e3f837d776b"
server_key_iv: "cb626ffe3970e917b0463e3f091c7cbc"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.551127 23537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:46.554472 23551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.556280 23537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:46.556538 23537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "d74ebf24dff1417ba4ed4f6b710f52e6"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "909c18474a65b944f83b6e3f837d776b"
server_key_iv: "cb626ffe3970e917b0463e3f091c7cbc"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.556699 23537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:46.571167 23537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:46.572047 23537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:46.572285 23537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:46.573000 23537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:46.574070 23537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:46.574147 23537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.574226 23537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:46.574268 23537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.585449 23537 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:43629
I20260928 14:46:46.585470 23664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:43629 every 8 connection(s)
I20260928 14:46:46.586516 23537 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:46.588900 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23537
I20260928 14:46:46.589062 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:46.589500 14205 external_mini_cluster.cc:1428] Setting key bab6326d604f936ed2114415a9575d41
I20260928 14:46:46.592185 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--builtin_ntp_servers=127.13.223.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:46:46.600023 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.588994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:43277 (local address 127.13.223.126:41129)
0928 14:46:46.589203 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:46.589209 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:46.590584 (+ 1375us) server_negotiation.cc:407] Connection header received
0928 14:46:46.592591 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:46.592594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:46.592649 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:46.592731 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:46.594125 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.595063 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.597384 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.597623 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.597717 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:46.599458 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:46.599461 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:46.599464 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:46.599479 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:46.599501 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:46.599510 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:46.599585 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:46.599813 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 14:46:46.599890 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260928 14:46:46.601559 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.589455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41129 (local address 127.13.223.65:43277)
0928 14:46:46.590391 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:46.590444 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:46.592257 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:46.592948 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:46.592961 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:46.593096 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:46.593933 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:46.593953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.595236 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.595242 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:46.597161 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:46.597182 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.597840 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.597846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:46.598133 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:46.599012 (+ 879us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:46.599016 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:46.599091 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:46.599213 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:46.599614 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:46.599627 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:46.599635 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:46:46.599814 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 14:46:46.600225 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":191,"threads_started":1}
I20260928 14:46:46.603226 23665 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41129
I20260928 14:46:46.603590 23665 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:46.604300 23665 heartbeater.cc:511] Master 127.13.223.126:41129 requested a full tablet report, sending...
I20260928 14:46:46.606307 23477 ts_manager.cc:194] Registered new tserver with Master: d74ebf24dff1417ba4ed4f6b710f52e6 (127.13.223.65:43629)
I20260928 14:46:46.607695 23477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.65:43277
W20260928 14:46:46.715313 23668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:46.715658 23668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:46.715744 23668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:46.720170 23668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:46.720295 23668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:46.720398 23668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:46.725631 23668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:46.727038 23668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:46.729324 23668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:46:46.741786 23668 server_base.cc:1034] running on GCE node
W20260928 14:46:46.741848 23676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.741847 23674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.742174 23673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:46.742785 23668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:46.743636 23668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:46.745029 23668 hybrid_clock.cc:648] HybridClock initialized: now 1790606806745005 us; error 40 us; skew 500 ppm
I20260928 14:46:46.747496 23668 webserver.cc:460] Webserver started at http://127.13.223.66:34995/ using document root <none> and password file <none>
I20260928 14:46:46.748183 23668 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:46.748291 23668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:46.748550 23668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:46.750954 23668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "b8f5af4f52044c26b71ed02d0dd38818"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "398445886d6912876d6e29066df67f56"
server_key_iv: "1083e25863411c0a6d88e2f9fb20c90d"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.751585 23668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "b8f5af4f52044c26b71ed02d0dd38818"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "398445886d6912876d6e29066df67f56"
server_key_iv: "1083e25863411c0a6d88e2f9fb20c90d"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.755941 23668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:46.759024 23682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.760663 23668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:46:46.760825 23668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "b8f5af4f52044c26b71ed02d0dd38818"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "398445886d6912876d6e29066df67f56"
server_key_iv: "1083e25863411c0a6d88e2f9fb20c90d"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.760988 23668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:46.776413 23668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:46.777266 23668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:46.777482 23668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:46.778113 23668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:46.779219 23668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:46.779291 23668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.779366 23668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:46.779417 23668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.789208 23668 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:33875
I20260928 14:46:46.789228 23795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:33875 every 8 connection(s)
I20260928 14:46:46.790294 23668 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:46.791306 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23668
I20260928 14:46:46.791427 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:46.791739 14205 external_mini_cluster.cc:1428] Setting key 13ae6fa2474338ad4744032c47dc557c
I20260928 14:46:46.793983 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--builtin_ntp_servers=127.13.223.84:39497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 14:46:46.803323 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.792596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:45995 (local address 127.13.223.126:41129)
0928 14:46:46.792737 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:46.792742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:46.794130 (+ 1388us) server_negotiation.cc:407] Connection header received
0928 14:46:46.796189 (+ 2059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:46.796192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:46.796250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:46.796339 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:46.797601 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.798308 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.800566 (+ 2258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.800900 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.801032 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:46.802706 (+ 1674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:46.802709 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:46.802712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:46.802728 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:46.802750 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:46.802759 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:46.802839 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:46.803109 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 14:46:46.803180 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:46:46.804777 23798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.793136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41129 (local address 127.13.223.66:45995)
0928 14:46:46.793957 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:46.794027 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:46.795963 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:46.796530 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:46.796542 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:46.796663 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:46.797415 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:46.797438 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.798475 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.798481 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:46.800348 (+ 1867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:46.800366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:46.801132 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:46.801140 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:46.801445 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:46.802257 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:46.802260 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:46.802330 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:46.802445 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:46.802871 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:46.802884 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:46.802894 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:46:46.803055 (+ 161us) client_negotiation.cc:240] Negotiation successful
0928 14:46:46.803463 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":115,"threads_started":1}
I20260928 14:46:46.806331 23796 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41129
I20260928 14:46:46.806694 23796 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:46.807392 23796 heartbeater.cc:511] Master 127.13.223.126:41129 requested a full tablet report, sending...
I20260928 14:46:46.808857 23477 ts_manager.cc:194] Registered new tserver with Master: b8f5af4f52044c26b71ed02d0dd38818 (127.13.223.66:33875)
I20260928 14:46:46.809497 23477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.66:45995
W20260928 14:46:46.918695 23799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:46.919057 23799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:46.919130 23799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:46.923749 23799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:46.923866 23799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:46.923974 23799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:46.929384 23799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:46.930785 23799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:46.932284 23799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:46.943059 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.943118 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:46.943060 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:46.943434 23799 server_base.cc:1034] running on GCE node
I20260928 14:46:46.943974 23799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:46.944734 23799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:46.945953 23799 hybrid_clock.cc:648] HybridClock initialized: now 1790606806945930 us; error 37 us; skew 500 ppm
I20260928 14:46:46.948546 23799 webserver.cc:460] Webserver started at http://127.13.223.67:36109/ using document root <none> and password file <none>
I20260928 14:46:46.949199 23799 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:46.949301 23799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:46.949548 23799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:46.951973 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "d5da475ac8e647459c78138471d90868"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "f7241348b7daed5e2b4acc2854b6459e"
server_key_iv: "9d8ecf0a1dd47168a60a9e8f929a930c"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.952598 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "d5da475ac8e647459c78138471d90868"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "f7241348b7daed5e2b4acc2854b6459e"
server_key_iv: "9d8ecf0a1dd47168a60a9e8f929a930c"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.956830 23799 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:46:46.960058 23813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.961753 23799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:46:46.961921 23799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "d5da475ac8e647459c78138471d90868"
format_stamp: "Formatted at 2026-09-28 14:46:46 on dist-test-slave-gkhx"
server_key: "f7241348b7daed5e2b4acc2854b6459e"
server_key_iv: "9d8ecf0a1dd47168a60a9e8f929a930c"
server_key_version: "encryptionkey@0"
I20260928 14:46:46.962092 23799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:46.979609 23799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:46.980458 23799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:46.980680 23799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:46.981324 23799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:46.982365 23799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:46.982439 23799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:46.982520 23799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:46.982589 23799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260928 14:46:46.992291 23799 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:44019
I20260928 14:46:46.992374 23926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:44019 every 8 connection(s)
I20260928 14:46:46.993710 23799 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:47.002228 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23799
I20260928 14:46:47.002346 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:47.002859 14205 external_mini_cluster.cc:1428] Setting key dd0e39629df0c7740160e6027e9c6fb4
I20260928 14:46:47.005919 23516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.996404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:39523 (local address 127.13.223.126:41129)
0928 14:46:46.996545 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:46.996549 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:46.997877 (+ 1328us) server_negotiation.cc:407] Connection header received
0928 14:46:46.999557 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:46.999561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:46.999632 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:46.999715 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.000790 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:47.001381 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:47.002924 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:47.003229 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:47.003361 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:47.004769 (+ 1408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:47.004772 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:47.004775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:47.004787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:47.005096 (+ 309us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:47.005124 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:47.005256 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:47.005668 (+ 412us) server_negotiation.cc:299] Negotiation successful
0928 14:46:47.005765 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:46:47.007063 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:46.996866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41129 (local address 127.13.223.67:39523)
0928 14:46:46.997608 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:46.997654 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:46.999289 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:46.999929 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:46.999938 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:47.000046 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:47.000613 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:47.000628 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:47.001539 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:47.001550 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:47.002774 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:47.002786 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:47.003426 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:47.003430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:47.003659 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:47.004429 (+ 770us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:47.004430 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:47.004480 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:47.004590 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:47.005427 (+ 837us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:47.005439 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:47.005447 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 14:46:47.005668 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 14:46:47.006044 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":132,"threads_started":1}
I20260928 14:46:47.008545 23927 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41129
I20260928 14:46:47.008875 23927 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:47.009570 23927 heartbeater.cc:511] Master 127.13.223.126:41129 requested a full tablet report, sending...
I20260928 14:46:47.010993 23477 ts_manager.cc:194] Registered new tserver with Master: d5da475ac8e647459c78138471d90868 (127.13.223.67:44019)
I20260928 14:46:47.011626 23477 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.67:39523
I20260928 14:46:47.018307 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:46:47.021751 23516 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:47.020643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.13.223.126:41129)
0928 14:46:47.020877 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.020883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.020993 (+ 110us) server_negotiation.cc:407] Connection header received
0928 14:46:47.021171 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.021175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.021234 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.021332 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.021622 (+ 290us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47590: BlockingRecv error: recv got EOF from 127.0.0.1:47590 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20260928 14:46:47.022224 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23537
I20260928 14:46:47.029804 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23668
I20260928 14:46:47.036033 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23799
I20260928 14:46:47.041903 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23446
2026-09-28T14:46:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (989 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T14:46:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:47Z Disabled control of system clock
I20260928 14:46:47.074015 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43993
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:43993
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:46:47.196563 23938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:47.196925 23938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:47.196966 23938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:47.201347 23938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:47.201458 23938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:47.201474 23938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:47.201493 23938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:47.201508 23938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:47.206790 23938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:43993
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:43993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:47.208170 23938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:47.209728 23938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:47.220232 23943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.220198 23944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.220198 23946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:47.221086 23938 server_base.cc:1034] running on GCE node
I20260928 14:46:47.221861 23938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:47.223238 23938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:47.224437 23938 hybrid_clock.cc:648] HybridClock initialized: now 1790606807224422 us; error 34 us; skew 500 ppm
I20260928 14:46:47.226723 23938 webserver.cc:460] Webserver started at http://127.13.223.126:41293/ using document root <none> and password file <none>
I20260928 14:46:47.227397 23938 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:47.227470 23938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:47.227705 23938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:47.229990 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "985789753619462f9b45ddd6bf324fdd"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "c4a6a9d250b2253d39a9c252e79d67ab"
server_key_iv: "5ae98ae6d50fcb87b11d88a9b7a1a88f"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.230706 23938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "985789753619462f9b45ddd6bf324fdd"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "c4a6a9d250b2253d39a9c252e79d67ab"
server_key_iv: "5ae98ae6d50fcb87b11d88a9b7a1a88f"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.235651 23938 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.002s
I20260928 14:46:47.239311 23952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.241422 23938 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:47.241640 23938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "985789753619462f9b45ddd6bf324fdd"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "c4a6a9d250b2253d39a9c252e79d67ab"
server_key_iv: "5ae98ae6d50fcb87b11d88a9b7a1a88f"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.241895 23938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:47.247378 23938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:47.248158 23938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:47.248387 23938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:47.257514 23938 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:43993
I20260928 14:46:47.257519 24004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:43993 every 8 connection(s)
I20260928 14:46:47.258874 23938 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:47.261565 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 23938
I20260928 14:46:47.261693 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:47.262008 14205 external_mini_cluster.cc:1428] Setting key ee8c83f87a980f171383e878cdb74d81
I20260928 14:46:47.263201 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:47.268999 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.263758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.13.223.126:43993)
0928 14:46:47.264550 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.264566 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.264608 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:46:47.265375 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.265396 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.265845 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.266276 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.267531 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:47.267536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:47.267550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:47.267562 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:47.267624 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:47.267644 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:47.267809 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:47.267875 (+ 66us) server_negotiation.cc:299] Negotiation successful
0928 14:46:47.268207 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":162,"threads_started":1}
I20260928 14:46:47.271027 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd: Bootstrap starting.
I20260928 14:46:47.273522 24005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:47.274374 24005 log.cc:824] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:47.277074 24005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd: No bootstrap required, opened a new log
I20260928 14:46:47.281129 24005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } }
I20260928 14:46:47.281462 24005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:47.281534 24005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 985789753619462f9b45ddd6bf324fdd, State: Initialized, Role: FOLLOWER
I20260928 14:46:47.281983 24005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } }
I20260928 14:46:47.282143 24005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:47.282243 24005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:47.282373 24005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:47.283473 24005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } }
I20260928 14:46:47.283862 24005 leader_election.cc:302] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 985789753619462f9b45ddd6bf324fdd; no voters:
I20260928 14:46:47.284247 24005 leader_election.cc:288] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:47.284456 24010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:47.284700 24010 raft_consensus.cc:686] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [term 1 LEADER]: Becoming Leader. State: Replica: 985789753619462f9b45ddd6bf324fdd, State: Running, Role: LEADER
I20260928 14:46:47.285053 24010 consensus_queue.cc:237] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } }
I20260928 14:46:47.285609 24005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:47.286860 24012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 985789753619462f9b45ddd6bf324fdd. Latest consensus state: current_term: 1 leader_uuid: "985789753619462f9b45ddd6bf324fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } } }
I20260928 14:46:47.287014 24012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:47.287472 24019 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:47.288295 24011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "985789753619462f9b45ddd6bf324fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "985789753619462f9b45ddd6bf324fdd" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 43993 } } }
I20260928 14:46:47.288415 24011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:47.291260 24019 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:47.298008 24019 catalog_manager.cc:1347] Generated new cluster ID: 8d972b7fe3e744f4b3f606674ec48bba
I20260928 14:46:47.298107 24019 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:47.313268 24019 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:47.313687 24019 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:47.319734 24019 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 985789753619462f9b45ddd6bf324fdd: Generated new TSK 0
I20260928 14:46:47.320595 24019 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:46:47.323944 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--builtin_ntp_servers=127.13.223.84:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 14:46:47.459632 24029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:47.459995 24029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:47.460039 24029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:47.465111 24029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:47.465204 24029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:47.465269 24029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:47.470799 24029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:47.472059 24029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:47.473461 24029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:47.484220 24037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.484251 24034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.484480 24035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:47.484963 24029 server_base.cc:1034] running on GCE node
I20260928 14:46:47.485582 24029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:47.486434 24029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:47.487603 24029 hybrid_clock.cc:648] HybridClock initialized: now 1790606807487578 us; error 52 us; skew 500 ppm
I20260928 14:46:47.489657 24029 webserver.cc:460] Webserver started at http://127.13.223.65:45491/ using document root <none> and password file <none>
I20260928 14:46:47.490250 24029 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:47.490322 24029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:47.490516 24029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:47.492869 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "f8277cdeafab4c158041c4ad197e43bc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "bfc889c7f58b865dd9e1bd43d715d345"
server_key_iv: "4b1f32ec433d52311f5486a5adf60f56"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.493451 24029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "f8277cdeafab4c158041c4ad197e43bc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "bfc889c7f58b865dd9e1bd43d715d345"
server_key_iv: "4b1f32ec433d52311f5486a5adf60f56"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.497895 24029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 14:46:47.501240 24043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.502952 24029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:47.503134 24029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "f8277cdeafab4c158041c4ad197e43bc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "bfc889c7f58b865dd9e1bd43d715d345"
server_key_iv: "4b1f32ec433d52311f5486a5adf60f56"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.503307 24029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:47.508607 24029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:47.509406 24029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:47.509608 24029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:47.510226 24029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:47.511327 24029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:47.511374 24029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.511413 24029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:47.511428 24029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.520675 24029 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:43603
I20260928 14:46:47.520701 24156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:43603 every 8 connection(s)
I20260928 14:46:47.521740 24029 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:47.522599 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24029
I20260928 14:46:47.522727 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:47.523079 14205 external_mini_cluster.cc:1428] Setting key 95e2a3eddfa1ac77f3cb9769fd3ff96f
I20260928 14:46:47.525626 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--builtin_ntp_servers=127.13.223.84:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:46:47.530437 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.524110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:60071 (local address 127.13.223.126:43993)
0928 14:46:47.524368 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.524374 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.525979 (+ 1605us) server_negotiation.cc:407] Connection header received
0928 14:46:47.527970 (+ 1991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.527974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.528025 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.528116 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.529674 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:47.529678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:47.529680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:47.529696 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:47.529720 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:47.529730 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:47.529853 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:47.530207 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 14:46:47.530277 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260928 14:46:47.531976 24159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.524697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43993 (local address 127.13.223.65:60071)
0928 14:46:47.525726 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:47.525799 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:47.527702 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:47.528375 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:47.528388 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:47.528530 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:47.529264 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:47.529269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:47.529343 (+ 74us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:47.529453 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:47.529930 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:47.529944 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:47.529954 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:46:47.530248 (+ 294us) client_negotiation.cc:240] Negotiation successful
0928 14:46:47.530645 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":167,"threads_started":1}
I20260928 14:46:47.534466 24157 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43993
I20260928 14:46:47.534829 24157 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:47.535353 24157 heartbeater.cc:511] Master 127.13.223.126:43993 requested a full tablet report, sending...
I20260928 14:46:47.537786 23969 ts_manager.cc:194] Registered new tserver with Master: f8277cdeafab4c158041c4ad197e43bc (127.13.223.65:43603)
W20260928 14:46:47.652909 24160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:47.653228 24160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:47.653271 24160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:47.658077 24160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:47.658182 24160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:47.658248 24160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:47.663802 24160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24160
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:47.665089 24160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:47.666604 24160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:47.677575 24165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.677575 24166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.677609 24168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:47.678086 24160 server_base.cc:1034] running on GCE node
I20260928 14:46:47.678632 24160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:47.679548 24160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:47.680799 24160 hybrid_clock.cc:648] HybridClock initialized: now 1790606807680770 us; error 46 us; skew 500 ppm
I20260928 14:46:47.683105 24160 webserver.cc:460] Webserver started at http://127.13.223.66:32903/ using document root <none> and password file <none>
I20260928 14:46:47.683826 24160 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:47.683907 24160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:47.684098 24160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:47.686511 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "2448fbd7b17645578eb87431db3a48cc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "558e442009b1b067f6bc726dc13f90e1"
server_key_iv: "dedb1a9449746ad8b8e3762c32962af0"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.687325 24160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "2448fbd7b17645578eb87431db3a48cc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "558e442009b1b067f6bc726dc13f90e1"
server_key_iv: "dedb1a9449746ad8b8e3762c32962af0"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.691932 24160 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:47.695395 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.697256 24160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:47.697427 24160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "2448fbd7b17645578eb87431db3a48cc"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "558e442009b1b067f6bc726dc13f90e1"
server_key_iv: "dedb1a9449746ad8b8e3762c32962af0"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.697561 24160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:47.703049 24160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:47.703826 24160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:47.703996 24160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:47.704550 24160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:47.705510 24160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:47.705559 24160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.705600 24160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:47.705614 24160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.715389 24160 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:40047
I20260928 14:46:47.715492 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:40047 every 8 connection(s)
I20260928 14:46:47.716475 24160 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:47.723899 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.718695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:38075 (local address 127.13.223.126:43993)
0928 14:46:47.718907 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.718911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.720191 (+ 1280us) server_negotiation.cc:407] Connection header received
0928 14:46:47.721642 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.721645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.721694 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.721796 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.723354 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:47.723359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:47.723369 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:47.723383 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:47.723406 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:47.723416 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:47.723531 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:47.723692 (+ 161us) server_negotiation.cc:299] Negotiation successful
0928 14:46:47.723749 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 14:46:47.724432 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24160
I20260928 14:46:47.724597 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:47.724948 14205 external_mini_cluster.cc:1428] Setting key 7fa46e0a239b9a4ddc965847eb15bacb
I20260928 14:46:47.725126 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.719206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43993 (local address 127.13.223.66:38075)
0928 14:46:47.720013 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:47.720061 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:47.721392 (+ 1331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:47.722012 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:47.722019 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:47.722121 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:47.723006 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:47.723008 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:47.723066 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:47.723149 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:47.723516 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:47.723526 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:47.723532 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 14:46:47.723820 (+ 288us) client_negotiation.cc:240] Negotiation successful
0928 14:46:47.724117 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":131,"threads_started":1}
I20260928 14:46:47.726675 24288 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43993
I20260928 14:46:47.726979 24288 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:47.727411 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--builtin_ntp_servers=127.13.223.84:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 14:46:47.727447 24288 heartbeater.cc:511] Master 127.13.223.126:43993 requested a full tablet report, sending...
I20260928 14:46:47.728798 23969 ts_manager.cc:194] Registered new tserver with Master: 2448fbd7b17645578eb87431db3a48cc (127.13.223.66:40047)
W20260928 14:46:47.851227 24291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:47.851557 24291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:47.851599 24291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:47.855881 24291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:47.855957 24291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:47.856019 24291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:47.861191 24291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:43993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:47.862432 24291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:47.863991 24291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:47.874616 24296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.874610 24299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:47.874610 24297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:47.875145 24291 server_base.cc:1034] running on GCE node
I20260928 14:46:47.875644 24291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:47.876415 24291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:47.877636 24291 hybrid_clock.cc:648] HybridClock initialized: now 1790606807877621 us; error 29 us; skew 500 ppm
I20260928 14:46:47.880151 24291 webserver.cc:460] Webserver started at http://127.13.223.67:46397/ using document root <none> and password file <none>
I20260928 14:46:47.880856 24291 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:47.880964 24291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:47.881213 24291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:47.883729 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "488a323393c5427d9110df1247f032b0"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "38954fa47e16d8c2d3051e3de3f7302f"
server_key_iv: "8ede5e6cbdd4b0d49f0c8891b43d86ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.884426 24291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "488a323393c5427d9110df1247f032b0"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "38954fa47e16d8c2d3051e3de3f7302f"
server_key_iv: "8ede5e6cbdd4b0d49f0c8891b43d86ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.889256 24291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:46:47.892577 24305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.894207 24291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:46:47.894410 24291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "488a323393c5427d9110df1247f032b0"
format_stamp: "Formatted at 2026-09-28 14:46:47 on dist-test-slave-gkhx"
server_key: "38954fa47e16d8c2d3051e3de3f7302f"
server_key_iv: "8ede5e6cbdd4b0d49f0c8891b43d86ae"
server_key_version: "encryptionkey@0"
I20260928 14:46:47.894598 24291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:47.900468 24291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:47.901562 24291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:47.901774 24291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:47.902540 24291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:47.903759 24291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:47.903842 24291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.903921 24291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:47.903957 24291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:47.914407 24291 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:36599
I20260928 14:46:47.914430 24418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:36599 every 8 connection(s)
I20260928 14:46:47.915506 24291 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:47.916240 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24291
I20260928 14:46:47.916369 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:47.916725 14205 external_mini_cluster.cc:1428] Setting key 12bf658e543cf2e8f92f3417c9dd1a05
I20260928 14:46:47.923010 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.917662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:56623 (local address 127.13.223.126:43993)
0928 14:46:47.917801 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.917804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.919387 (+ 1583us) server_negotiation.cc:407] Connection header received
0928 14:46:47.920761 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.920764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.920858 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.920957 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.922234 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:47.922237 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:47.922240 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:46:47.922252 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:47.922276 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:46:47.922284 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:47.922400 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:47.922824 (+ 424us) server_negotiation.cc:299] Negotiation successful
0928 14:46:47.922879 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:46:47.923976 24421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:47.918280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:43993 (local address 127.13.223.67:56623)
0928 14:46:47.919209 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:47.919271 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:47.920496 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:47.921132 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:47.921140 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:47.921241 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:47.921948 (+ 707us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:46:47.921949 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:47.921995 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:46:47.922072 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:47.922428 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:47.922447 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:47.922454 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 14:46:47.922847 (+ 393us) client_negotiation.cc:240] Negotiation successful
0928 14:46:47.923113 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":189,"threads_started":1}
I20260928 14:46:47.925366 24419 heartbeater.cc:347] Connected to a master server at 127.13.223.126:43993
I20260928 14:46:47.925644 24419 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:47.926111 24419 heartbeater.cc:511] Master 127.13.223.126:43993 requested a full tablet report, sending...
I20260928 14:46:47.927371 23969 ts_manager.cc:194] Registered new tserver with Master: 488a323393c5427d9110df1247f032b0 (127.13.223.67:36599)
I20260928 14:46:47.931859 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 14:46:47.935323 24008 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:46:47.934200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40918 (local address 127.13.223.126:43993)
0928 14:46:47.934419 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:47.934423 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:47.934540 (+ 117us) server_negotiation.cc:407] Connection header received
0928 14:46:47.934746 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:47.934749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:47.934794 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:47.934889 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:47.935143 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40918: BlockingRecv error: recv got EOF from 127.0.0.1:40918 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 14:46:47.935582 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24029
I20260928 14:46:47.942528 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24160
I20260928 14:46:47.948661 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24291
I20260928 14:46:47.954325 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 23938
2026-09-28T14:46:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (912 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:48 dist-test-slave-gkhx krb5kdc[24429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:48 dist-test-slave-gkhx krb5kdc[24429](info): set up 2 sockets
Sep 28 14:46:48 dist-test-slave-gkhx krb5kdc[24429](info): commencing operation
krb5kdc: starting...
W20260928 14:46:49.121858 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:49 dist-test-slave-gkhx krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:46:49.267071 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35195
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:35195
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:49.390282 24442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:49.390754 24442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:49.390859 24442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:49.395327 24442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:46:49.395432 24442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:49.395467 24442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:49.395507 24442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:46:49.395560 24442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:46:49.401732 24442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:35195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:49.403331 24442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:49.404500 24442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:49.414692 24450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:49.414768 24442 server_base.cc:1034] running on GCE node
W20260928 14:46:49.414691 24447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:49.414793 24448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:49.415650 24442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:49.416923 24442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:49.418198 24442 hybrid_clock.cc:648] HybridClock initialized: now 1790606809418182 us; error 44 us; skew 500 ppm
Sep 28 14:46:49 dist-test-slave-gkhx krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606809, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:49.421697 24442 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:46:49.422896 24442 webserver.cc:460] Webserver started at http://127.13.223.126:44275/ using document root <none> and password file <none>
I20260928 14:46:49.423521 24442 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:49.423616 24442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:49.423866 24442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:49.426218 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "38b048d1a3bd43a896cebdf60f0bffd1"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "8d8f4d37b31342b92f9361d2dc3d159a"
server_key_iv: "0e53a805347dfcf76baf26e1fe469ae1"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.426926 24442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "38b048d1a3bd43a896cebdf60f0bffd1"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "8d8f4d37b31342b92f9361d2dc3d159a"
server_key_iv: "0e53a805347dfcf76baf26e1fe469ae1"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.431692 24442 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:46:49.435127 24457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:49.436691 24442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:49.436882 24442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "38b048d1a3bd43a896cebdf60f0bffd1"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "8d8f4d37b31342b92f9361d2dc3d159a"
server_key_iv: "0e53a805347dfcf76baf26e1fe469ae1"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.437031 24442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:49.453169 24442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:49.457319 24442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:49.457626 24442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:49.465917 24442 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:35195
I20260928 14:46:49.466221 24509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:35195 every 8 connection(s)
I20260928 14:46:49.467154 24442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:46:49.470377 24510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:49.475878 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24442
I20260928 14:46:49.475996 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:46:49.476379 14205 external_mini_cluster.cc:1428] Setting key a7a5671d9939689305b94bf8f6173fb0
I20260928 14:46:49.477481 24510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: Bootstrap starting.
I20260928 14:46:49.480638 24510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:49.481701 24510 log.cc:824] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:49.484328 24510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: No bootstrap required, opened a new log
Sep 28 14:46:49 dist-test-slave-gkhx krb5kdc[24429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:49.489571 24510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } }
I20260928 14:46:49.489995 24510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:49.490067 24510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b048d1a3bd43a896cebdf60f0bffd1, State: Initialized, Role: FOLLOWER
I20260928 14:46:49.490752 24510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } }
I20260928 14:46:49.490970 24510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:46:49.491039 24510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:46:49.491155 24510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:49.492553 24510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } }
I20260928 14:46:49.493005 24510 leader_election.cc:302] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b048d1a3bd43a896cebdf60f0bffd1; no voters:
I20260928 14:46:49.493426 24510 leader_election.cc:288] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:46:49.493628 24515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:49.493893 24515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [term 1 LEADER]: Becoming Leader. State: Replica: 38b048d1a3bd43a896cebdf60f0bffd1, State: Running, Role: LEADER
I20260928 14:46:49.494359 24515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } }
I20260928 14:46:49.494971 24510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:46:49.495725 24515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38b048d1a3bd43a896cebdf60f0bffd1. Latest consensus state: current_term: 1 leader_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } } }
I20260928 14:46:49.495817 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:49.478096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.13.223.126:35195)
0928 14:46:49.478651 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:49.478673 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:49.478727 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:46:49.479744 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:49.479785 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:49.480296 (+ 511us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:49.480768 (+ 472us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:49.481619 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.483276 (+ 1657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.484303 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.485106 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.485312 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:49.488959 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:49.488968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:49.488992 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:49.489037 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:49.491618 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:49.492191 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:49.492198 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:49.492205 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:49.492398 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:49.492786 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:49.492790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:49.492792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:49.493433 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:49.493642 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:49.494239 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 14:46:49.494649 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":145,"threads_started":1}
I20260928 14:46:49.495972 24515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:49.496371 24516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b048d1a3bd43a896cebdf60f0bffd1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35195 } } }
I20260928 14:46:49.496541 24516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:46:49.496368 24519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:46:49.500415 24519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:46:49.507899 24519 catalog_manager.cc:1347] Generated new cluster ID: 1880297aba1e4c63bfe1359daa1bd6c8
I20260928 14:46:49.508023 24519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:46:49.530118 24519 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:46:49.531275 24519 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:46:49.536005 24519 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: Generated new TSK 0
I20260928 14:46:49.536849 24519 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:46:49.578109 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:49.728053 24536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:49.728448 24536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:49.728530 24536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:49.732889 24536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:49.733027 24536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:49.733134 24536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:46:49.739146 24536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:49.740731 24536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:49.742126 24536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:49.753705 24544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:49.753906 24542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:49.753775 24541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:49.753945 24536 server_base.cc:1034] running on GCE node
I20260928 14:46:49.754693 24536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:49.755488 24536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:49.756728 24536 hybrid_clock.cc:648] HybridClock initialized: now 1790606809756714 us; error 41 us; skew 500 ppm
Sep 28 14:46:49 dist-test-slave-gkhx krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606809, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:49.766146 24536 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:46:49.767445 24536 webserver.cc:460] Webserver started at http://127.13.223.65:33515/ using document root <none> and password file <none>
I20260928 14:46:49.768151 24536 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:49.768265 24536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:49.768548 24536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:49.771095 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "5cbd3d5a4932414683c8544a636b7b28"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "a1c3b6a2ea02732632e435775fd64101"
server_key_iv: "7bd32b8b581afb923b5f699288c9643c"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.771772 24536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "5cbd3d5a4932414683c8544a636b7b28"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "a1c3b6a2ea02732632e435775fd64101"
server_key_iv: "7bd32b8b581afb923b5f699288c9643c"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.776993 24536 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 14:46:49.780295 24551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:49.781756 24536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:46:49.781951 24536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "5cbd3d5a4932414683c8544a636b7b28"
format_stamp: "Formatted at 2026-09-28 14:46:49 on dist-test-slave-gkhx"
server_key: "a1c3b6a2ea02732632e435775fd64101"
server_key_iv: "7bd32b8b581afb923b5f699288c9643c"
server_key_version: "encryptionkey@0"
I20260928 14:46:49.782105 24536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:49.810163 24536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:49.814489 24536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:49.814779 24536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:49.815480 24536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:49.816567 24536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:49.816649 24536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:49.816731 24536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:49.816789 24536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:49.828545 24536 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:39033
I20260928 14:46:49.828565 24664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:39033 every 8 connection(s)
I20260928 14:46:49.829679 24536 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:46:49.838178 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24536
I20260928 14:46:49.838327 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:46:49.838706 14205 external_mini_cluster.cc:1428] Setting key 8be99c88c028590c18ce1f5d75fc6b2b
Sep 28 14:46:49 dist-test-slave-gkhx krb5kdc[24429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606809, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:49.846767 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:49.831884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:59449 (local address 127.13.223.126:35195)
0928 14:46:49.832042 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:49.832046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:49.833078 (+ 1032us) server_negotiation.cc:407] Connection header received
0928 14:46:49.834652 (+ 1574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:49.834656 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:49.834715 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:49.834803 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:49.836269 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.836932 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.838214 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.838670 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.838794 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:49.842063 (+ 3269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:49.842070 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:49.842077 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:49.842142 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:49.843854 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:49.844559 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:49.844566 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:49.844571 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:49.844680 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:49.845212 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:49.845219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:49.845224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:49.845540 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:49.845662 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:49.846421 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 14:46:49.846604 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:46:49.848485 24667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:49.832343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35195 (local address 127.13.223.65:59449)
0928 14:46:49.832894 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:49.832944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:49.834333 (+ 1389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:49.835084 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:49.835093 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:49.835534 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:49.836092 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:49.836107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.837076 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.837079 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:49.838065 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:49.838075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:49.838975 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:49.838979 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:49.839200 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:49.839961 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:49.839983 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:49.841871 (+ 1888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:49.844030 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:49.844036 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:49.844081 (+ 45us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:49.844425 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:49.844892 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:49.844896 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:49.844897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:49.845078 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:49.845726 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:49.845733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:49.846067 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 14:46:49.846318 (+ 251us) client_negotiation.cc:240] Negotiation successful
0928 14:46:49.846913 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":123,"threads_started":1}
I20260928 14:46:49.850441 24665 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35195
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:46:49.850792 24665 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:49.851506 24665 heartbeater.cc:511] Master 127.13.223.126:35195 requested a full tablet report, sending...
I20260928 14:46:49.853684 24474 ts_manager.cc:194] Registered new tserver with Master: 5cbd3d5a4932414683c8544a636b7b28 (127.13.223.65:39033)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:46:49.855504 24474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:59449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:46:49.875497 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:50.003607 24670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:50.003998 24670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:50.004093 24670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:50.010346 24670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:50.010471 24670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:50.010610 24670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:46:50.016953 24670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:50.018178 24670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:50.019347 24670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:50.031306 24676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:50.031252 24678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:50.031349 24675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:50.031427 24670 server_base.cc:1034] running on GCE node
I20260928 14:46:50.032130 24670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:50.033038 24670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:50.034269 24670 hybrid_clock.cc:648] HybridClock initialized: now 1790606810034254 us; error 32 us; skew 500 ppm
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:50.037793 24670 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:46:50.039057 24670 webserver.cc:460] Webserver started at http://127.13.223.66:42677/ using document root <none> and password file <none>
I20260928 14:46:50.039785 24670 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:50.039898 24670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:50.040174 24670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:50.042920 24670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "de13ec7001324f62969e3513b78105ff"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "d0e3ac26a5165c63cc53a304676ee938"
server_key_iv: "d73088d146c36a3b85c32804a786a6b3"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.043597 24670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "de13ec7001324f62969e3513b78105ff"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "d0e3ac26a5165c63cc53a304676ee938"
server_key_iv: "d73088d146c36a3b85c32804a786a6b3"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.048210 24670 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:46:50.051726 24685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.053383 24670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:46:50.053551 24670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "de13ec7001324f62969e3513b78105ff"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "d0e3ac26a5165c63cc53a304676ee938"
server_key_iv: "d73088d146c36a3b85c32804a786a6b3"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.053695 24670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:50.087735 24670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:50.091401 24670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:50.091638 24670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:50.092276 24670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:50.093165 24670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:50.093247 24670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.093348 24670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:50.093400 24670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.103281 24670 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:41909
I20260928 14:46:50.103308 24798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:41909 every 8 connection(s)
I20260928 14:46:50.104342 24670 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:46:50.114176 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24670
I20260928 14:46:50.114363 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:46:50.114753 14205 external_mini_cluster.cc:1428] Setting key fac9860c8f3c7649e679892e4d44c312
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:50.120764 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.106462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:38935 (local address 127.13.223.126:35195)
0928 14:46:50.106667 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.106672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.107613 (+ 941us) server_negotiation.cc:407] Connection header received
0928 14:46:50.109143 (+ 1530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.109147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.109200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.109298 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:50.111051 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.111620 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.112923 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.113168 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.113266 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.116593 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:50.116602 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:50.116604 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:50.116629 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:50.118264 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.119017 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.119020 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.119021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.119102 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.119491 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.119495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.119497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.119734 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:50.119852 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.120448 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.120592 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 14:46:50.121701 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.106936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35195 (local address 127.13.223.66:38935)
0928 14:46:50.107443 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.107490 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.108906 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.109523 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.109531 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.110006 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:50.110784 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.110804 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.111776 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.111779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.112782 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.112793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.113343 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.113347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.113513 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.114316 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:50.114379 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:50.116415 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:50.118478 (+ 2063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:50.118485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:50.118506 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:50.118889 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:50.119209 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:50.119212 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:50.119214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:50.119384 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:50.119852 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:50.119863 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:50.120162 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.120377 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.120753 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":113,"threads_started":1}
I20260928 14:46:50.123191 24799 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35195
I20260928 14:46:50.123503 24799 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:50.124163 24799 heartbeater.cc:511] Master 127.13.223.126:35195 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:46:50.125319 24474 ts_manager.cc:194] Registered new tserver with Master: de13ec7001324f62969e3513b78105ff (127.13.223.66:41909)
I20260928 14:46:50.126005 24474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:38935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:46:50.147619 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:46057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:46:50.276436 24804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:46:50.276759 24804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:46:50.276806 24804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:46:50.281250 24804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:46:50.281330 24804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:46:50.281415 24804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:46:50.287418 24804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:46057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35195
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:46:50.288612 24804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:46:50.289633 24804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:46:50.300596 24810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:50.300516 24809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:46:50.300563 24812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:46:50.300540 24804 server_base.cc:1034] running on GCE node
I20260928 14:46:50.301312 24804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:46:50.302078 24804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:46:50.303298 24804 hybrid_clock.cc:648] HybridClock initialized: now 1790606810303247 us; error 67 us; skew 500 ppm
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:46:50.306591 24804 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:46:50.307572 24804 webserver.cc:460] Webserver started at http://127.13.223.67:45387/ using document root <none> and password file <none>
I20260928 14:46:50.308138 24804 fs_manager.cc:359] Metadata directory not provided
I20260928 14:46:50.308204 24804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:46:50.308404 24804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:46:50.310688 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "dd5cca0c08384ee19660ec87004d9507"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "7f8ddc3535856f3f44a928e753804afd"
server_key_iv: "f070cc23522f39f8cf1044ee6faa7713"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.311354 24804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "dd5cca0c08384ee19660ec87004d9507"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "7f8ddc3535856f3f44a928e753804afd"
server_key_iv: "f070cc23522f39f8cf1044ee6faa7713"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.315569 24804 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:46:50.318341 24819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.319887 24804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:46:50.320005 24804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "dd5cca0c08384ee19660ec87004d9507"
format_stamp: "Formatted at 2026-09-28 14:46:50 on dist-test-slave-gkhx"
server_key: "7f8ddc3535856f3f44a928e753804afd"
server_key_iv: "f070cc23522f39f8cf1044ee6faa7713"
server_key_version: "encryptionkey@0"
I20260928 14:46:50.320169 24804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:46:50.349499 24804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:46:50.352998 24804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:46:50.353227 24804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:46:50.353919 24804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:46:50.354894 24804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:46:50.354952 24804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.355020 24804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:46:50.355054 24804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:46:50.364845 24804 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40351
I20260928 14:46:50.364863 24932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40351 every 8 connection(s)
I20260928 14:46:50.365854 24804 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:46:50.366240 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 24804
I20260928 14:46:50.366384 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:46:50.366753 14205 external_mini_cluster.cc:1428] Setting key 55a7f61f1faf45156e8302cd79aa60d7
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606810, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:46:50.382202 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.368113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:54497 (local address 127.13.223.126:35195)
0928 14:46:50.368276 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.368282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.369421 (+ 1139us) server_negotiation.cc:407] Connection header received
0928 14:46:50.370963 (+ 1542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.370966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.371013 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.371090 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:50.372294 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.372864 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.374231 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.374469 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.374583 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.377710 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:50.377715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:50.377717 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:50.377745 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:50.379401 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.380155 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.380158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.380160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.380301 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.380814 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.380818 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.380830 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.381123 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:50.381223 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.381956 (+ 733us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.382076 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:46:50.383262 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.368679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35195 (local address 127.13.223.67:54497)
0928 14:46:50.369246 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.369313 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.370742 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.371237 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.371244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.371633 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:46:50.372149 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.372164 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.373037 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.373040 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.374086 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.374097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.374827 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.374830 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.375004 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.375672 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:46:50.375691 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:46:50.377418 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:46:50.379557 (+ 2139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:50.379564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:50.379580 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:50.380027 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:50.380409 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:46:50.380412 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:46:50.380414 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:46:50.380612 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:46:50.381265 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:46:50.381273 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:46:50.381688 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.381884 (+ 196us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.382194 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":171,"threads_started":1}
I20260928 14:46:50.384649 24933 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35195
I20260928 14:46:50.385069 24933 heartbeater.cc:464] Registering TS with master...
I20260928 14:46:50.385723 24933 heartbeater.cc:511] Master 127.13.223.126:35195 requested a full tablet report, sending...
I20260928 14:46:50.387095 24474 ts_manager.cc:194] Registered new tserver with Master: dd5cca0c08384ee19660ec87004d9507 (127.13.223.67:40351)
I20260928 14:46:50.387699 24474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:54497
I20260928 14:46:50.394004 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:46:50.404785 24513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.396488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.13.223.126:35195)
0928 14:46:50.396674 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.396681 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.396865 (+ 184us) server_negotiation.cc:407] Connection header received
0928 14:46:50.396947 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.396950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.396994 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.397065 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:46:50.397900 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.398466 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.399502 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.399738 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.399886 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.401365 (+ 1479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:46:50.401368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:46:50.401370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:46:50.401395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:46:50.402966 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.403426 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.403428 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.403430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.403509 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:46:50.403962 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:46:50.403964 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:46:50.403966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:46:50.404207 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:46:50.404328 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.404550 (+ 222us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.404656 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:46:50.408317 24474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51878:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:46:50.411505 24474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:46:50.427685 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.421938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43644 (local address 127.13.223.65:39033)
0928 14:46:50.422751 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.422757 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.422780 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:46:50.422905 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.422909 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.423255 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.423448 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:50.423697 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424810 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.426226 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.427070 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.427178 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427340 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.427438 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.427537 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":68,"threads_started":1}
I20260928 14:46:50.427673 24944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.420942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41909 (local address 127.0.0.1:52650)
0928 14:46:50.421379 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.421431 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.421575 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.422635 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.422642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.422665 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:50.422838 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.422846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424325 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.424329 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.425694 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.425703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.426972 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.426975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.427096 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427162 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.427280 (+ 118us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.427488 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":4,"thread_start_us":111,"threads_started":1}
I20260928 14:46:50.427701 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.421763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:39033 (local address 127.0.0.1:43644)
0928 14:46:50.422224 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.422239 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.422322 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.423416 (+ 1094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.423419 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.423439 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:50.423570 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.423578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424996 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.425000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.426081 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.426091 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.427219 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.427225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.427313 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427335 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.427408 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.427506 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":30,"thread_start_us":94,"threads_started":1}
I20260928 14:46:50.428345 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.421284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60376 (local address 127.13.223.67:40351)
0928 14:46:50.421938 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.421952 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.421979 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:46:50.422151 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.422157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.422376 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.422577 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:50.423113 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424406 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.426322 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.427158 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.427263 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427398 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.428070 (+ 672us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.428189 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":178,"threads_started":1}
I20260928 14:46:50.428370 24947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.420942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40351 (local address 127.0.0.1:60376)
0928 14:46:50.421884 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.421897 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.421991 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.422780 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.422783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.422796 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:50.422968 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.422973 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424539 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.424542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.426180 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.426190 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.427468 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.427472 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.427562 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427583 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.427659 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.427731 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":844,"thread_start_us":110,"threads_started":1}
I20260928 14:46:50.431205 24867 tablet_service.cc:1467] Processing CreateTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f (DEFAULT_TABLE table=test-table [id=e27ebdeac0804a83be5a54723974bf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:50.431197 24599 tablet_service.cc:1467] Processing CreateTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f (DEFAULT_TABLE table=test-table [id=e27ebdeac0804a83be5a54723974bf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:50.432133 24945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.421087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52650 (local address 127.13.223.66:41909)
0928 14:46:50.422186 (+ 1099us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.422198 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.422218 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:46:50.422282 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.422287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.422443 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.423004 (+ 561us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:50.423171 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.424120 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.425846 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.426829 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.426949 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.427114 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.431782 (+ 4668us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.431947 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":988,"thread_start_us":138,"threads_started":1}
I20260928 14:46:50.432341 24599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1cb291cdd0444f5aa01d6cdcb32914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:50.432343 24867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1cb291cdd0444f5aa01d6cdcb32914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:50.435493 24733 tablet_service.cc:1467] Processing CreateTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f (DEFAULT_TABLE table=test-table [id=e27ebdeac0804a83be5a54723974bf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:46:50.436760 24733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1cb291cdd0444f5aa01d6cdcb32914f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:46:50.439373 24950 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Bootstrap starting.
I20260928 14:46:50.440737 24951 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: Bootstrap starting.
I20260928 14:46:50.441900 24950 tablet_bootstrap.cc:654] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:50.443203 24951 tablet_bootstrap.cc:654] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:50.443404 24950 log.cc:824] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:50.444069 24951 log.cc:824] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:50.445160 24954 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Bootstrap starting.
I20260928 14:46:50.446262 24951 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: No bootstrap required, opened a new log
I20260928 14:46:50.446336 24950 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: No bootstrap required, opened a new log
I20260928 14:46:50.446503 24951 ts_tablet_manager.cc:1397] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:46:50.446535 24950 ts_tablet_manager.cc:1397] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:46:50.448179 24954 tablet_bootstrap.cc:654] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Neither blocks nor log segments found. Creating new log.
I20260928 14:46:50.449148 24954 log.cc:824] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Log is configured to *not* fsync() on all Append() calls
I20260928 14:46:50.450373 24951 raft_consensus.cc:348] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.450708 24951 raft_consensus.cc:374] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:50.450764 24951 raft_consensus.cc:729] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cbd3d5a4932414683c8544a636b7b28, State: Initialized, Role: FOLLOWER
I20260928 14:46:50.450591 24950 raft_consensus.cc:348] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.450850 24950 raft_consensus.cc:374] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:50.451035 24950 raft_consensus.cc:729] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5cca0c08384ee19660ec87004d9507, State: Initialized, Role: FOLLOWER
I20260928 14:46:50.451280 24951 consensus_queue.cc:260] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.451494 24950 consensus_queue.cc:260] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.452315 24954 tablet_bootstrap.cc:492] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: No bootstrap required, opened a new log
I20260928 14:46:50.452417 24951 ts_tablet_manager.cc:1428] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260928 14:46:50.452491 24665 heartbeater.cc:503] Master 127.13.223.126:35195 was elected leader, sending a full tablet report...
I20260928 14:46:50.452548 24954 ts_tablet_manager.cc:1397] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:46:50.452912 24933 heartbeater.cc:503] Master 127.13.223.126:35195 was elected leader, sending a full tablet report...
I20260928 14:46:50.453088 24950 ts_tablet_manager.cc:1428] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 14:46:50.456542 24954 raft_consensus.cc:348] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.456938 24954 raft_consensus.cc:374] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:46:50.457024 24954 raft_consensus.cc:729] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de13ec7001324f62969e3513b78105ff, State: Initialized, Role: FOLLOWER
I20260928 14:46:50.457540 24954 consensus_queue.cc:260] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.458365 24799 heartbeater.cc:503] Master 127.13.223.126:35195 was elected leader, sending a full tablet report...
I20260928 14:46:50.458787 24954 ts_tablet_manager.cc:1428] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:46:50.492969 24957 raft_consensus.cc:482] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:46:50.493201 24957 raft_consensus.cc:504] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.494841 24957 leader_election.cc:288] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers de13ec7001324f62969e3513b78105ff (127.13.223.66:41909), 5cbd3d5a4932414683c8544a636b7b28 (127.13.223.65:39033)
I20260928 14:46:50.500887 24945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.495482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:59393 (local address 127.13.223.66:41909)
0928 14:46:50.495635 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.495641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.495777 (+ 136us) server_negotiation.cc:407] Connection header received
0928 14:46:50.496030 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.496036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.496099 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.496180 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:50.496706 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.497514 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.499505 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.500193 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.500346 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.500403 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.500668 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.500765 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:46:50.500865 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.495348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41909 (local address 127.13.223.67:59393)
0928 14:46:50.495636 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.495662 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.495831 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.496424 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.496427 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.496448 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:50.496570 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.496577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.497690 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.497694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.499293 (+ 1599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.499302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.500323 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.500327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.500446 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.500472 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.500559 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.500689 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":137}
I20260928 14:46:50.501505 24959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.495689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:39033 (local address 127.13.223.67:49735)
0928 14:46:50.496130 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.496147 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:46:50.496313 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:46:50.496730 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:46:50.496735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:46:50.496755 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:46:50.496864 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.496869 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.497826 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.497829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.499286 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:46:50.499302 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.500471 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.500475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:46:50.501119 (+ 644us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.501146 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:46:50.501240 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 14:46:50.501331 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":379,"mutex_wait_us":489,"thread_start_us":163,"threads_started":1}
I20260928 14:46:50.501566 24949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.495848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:49735 (local address 127.13.223.65:39033)
0928 14:46:50.496150 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.496157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.496255 (+ 98us) server_negotiation.cc:407] Connection header received
0928 14:46:50.496558 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.496564 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.496634 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.496710 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:46:50.496981 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.497720 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.499505 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.500196 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.500351 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.500403 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.501308 (+ 905us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.501432 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186}
I20260928 14:46:50.501900 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1cb291cdd0444f5aa01d6cdcb32914f" candidate_uuid: "dd5cca0c08384ee19660ec87004d9507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de13ec7001324f62969e3513b78105ff" is_pre_election: true
I20260928 14:46:50.502282 24753 raft_consensus.cc:2454] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5cca0c08384ee19660ec87004d9507 in term 0.
I20260928 14:46:50.502502 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1cb291cdd0444f5aa01d6cdcb32914f" candidate_uuid: "dd5cca0c08384ee19660ec87004d9507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbd3d5a4932414683c8544a636b7b28" is_pre_election: true
I20260928 14:46:50.502837 24823 leader_election.cc:302] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dd5cca0c08384ee19660ec87004d9507, de13ec7001324f62969e3513b78105ff; no voters:
I20260928 14:46:50.502882 24617 raft_consensus.cc:2454] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd5cca0c08384ee19660ec87004d9507 in term 0.
I20260928 14:46:50.503147 24957 raft_consensus.cc:2781] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:46:50.503278 24957 raft_consensus.cc:482] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:46:50.503358 24957 raft_consensus.cc:3037] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:50.504520 24957 raft_consensus.cc:504] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.504953 24957 leader_election.cc:288] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [CANDIDATE]: Term 1 election: Requested vote from peers de13ec7001324f62969e3513b78105ff (127.13.223.66:41909), 5cbd3d5a4932414683c8544a636b7b28 (127.13.223.65:39033)
I20260928 14:46:50.505501 24753 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1cb291cdd0444f5aa01d6cdcb32914f" candidate_uuid: "dd5cca0c08384ee19660ec87004d9507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "de13ec7001324f62969e3513b78105ff"
I20260928 14:46:50.505559 24617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1cb291cdd0444f5aa01d6cdcb32914f" candidate_uuid: "dd5cca0c08384ee19660ec87004d9507" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cbd3d5a4932414683c8544a636b7b28"
I20260928 14:46:50.505708 24617 raft_consensus.cc:3037] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:50.505771 24753 raft_consensus.cc:3037] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:46:50.507231 24617 raft_consensus.cc:2454] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5cca0c08384ee19660ec87004d9507 in term 1.
I20260928 14:46:50.507345 24753 raft_consensus.cc:2454] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd5cca0c08384ee19660ec87004d9507 in term 1.
I20260928 14:46:50.507644 24823 leader_election.cc:302] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cbd3d5a4932414683c8544a636b7b28, dd5cca0c08384ee19660ec87004d9507; no voters:
I20260928 14:46:50.508001 24957 raft_consensus.cc:2781] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:46:50.508270 24957 raft_consensus.cc:686] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 1 LEADER]: Becoming Leader. State: Replica: dd5cca0c08384ee19660ec87004d9507, State: Running, Role: LEADER
I20260928 14:46:50.508683 24957 consensus_queue.cc:237] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } }
I20260928 14:46:50.512156 24474 catalog_manager.cc:5495] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd5cca0c08384ee19660ec87004d9507 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "dd5cca0c08384ee19660ec87004d9507" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd5cca0c08384ee19660ec87004d9507" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40351 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 } health_report { overall_health: UNKNOWN } } }
I20260928 14:46:50.530205 24946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.526230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.13.223.67:40351)
0928 14:46:50.526357 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 14:46:50.526360 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:46:50.526373 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:46:50.526471 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:46:50.526473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:46:50.526531 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:46:50.526656 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:46:50.526971 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.527501 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.528443 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:46:50.528667 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:46:50.528766 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:46:50.529199 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:46:50.529707 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:46:50.529821 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:46:50.529997 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 14:46:50.530062 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 14:46:50.537676 24617 raft_consensus.cc:1261] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 1 FOLLOWER]: Refusing update from remote peer dd5cca0c08384ee19660ec87004d9507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:50.537675 24753 raft_consensus.cc:1261] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 1 FOLLOWER]: Refusing update from remote peer dd5cca0c08384ee19660ec87004d9507: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:46:50.538656 24957 consensus_queue.cc:1035] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "de13ec7001324f62969e3513b78105ff" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 41909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:50.538940 24960 consensus_queue.cc:1035] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cbd3d5a4932414683c8544a636b7b28" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:46:50.560544 24474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:50.560753 24474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51878:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:46:50.563652 24474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38b048d1a3bd43a896cebdf60f0bffd1: Sending DeleteTablet for 3 replicas of tablet d1cb291cdd0444f5aa01d6cdcb32914f
I20260928 14:46:50.564721 24733 tablet_service.cc:1514] Processing DeleteTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:52650
I20260928 14:46:50.564723 24599 tablet_service.cc:1514] Processing DeleteTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:43644
I20260928 14:46:50.564723 24867 tablet_service.cc:1514] Processing DeleteTablet for tablet d1cb291cdd0444f5aa01d6cdcb32914f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:46:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:60376
I20260928 14:46:50.565459 24974 tablet_replica.cc:330] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28: stopping tablet replica
I20260928 14:46:50.565917 24974 raft_consensus.cc:2229] T d1cb291cdd0444f5aa01d6cdcb32914f P 5cbd3d5a4932414683c8544a636b7b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:50.565981 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24536
I20260928 14:46:50.568002 24973 tablet_replica.cc:330] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: stopping tablet replica
I20260928 14:46:50.568400 24973 raft_consensus.cc:2229] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:46:50.568926 24973 raft_consensus.cc:2258] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:50.571302 24972 tablet_replica.cc:330] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: stopping tablet replica
I20260928 14:46:50.571738 24972 raft_consensus.cc:2229] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:46:50.572153 24972 raft_consensus.cc:2258] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:46:50.572412 24973 ts_tablet_manager.cc:1905] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:46:50.574074 24972 ts_tablet_manager.cc:1905] T d1cb291cdd0444f5aa01d6cdcb32914f P de13ec7001324f62969e3513b78105ff: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:46:50.574638 24461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:39033 (error 108)
I20260928 14:46:50.575784 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24670
I20260928 14:46:50.576305 24973 ts_tablet_manager.cc:1918] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:46:50.576737 24973 log.cc:1196] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/d1cb291cdd0444f5aa01d6cdcb32914f
I20260928 14:46:50.577234 24973 ts_tablet_manager.cc:1939] T d1cb291cdd0444f5aa01d6cdcb32914f P dd5cca0c08384ee19660ec87004d9507: Deleting consensus metadata
I20260928 14:46:50.577632 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.577208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:39033 (local address 127.0.0.1:43650)
0928 14:46:50.577452 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.577534 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:39033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260928 14:46:50.577842 24461 catalog_manager.cc:4584] TS 5cbd3d5a4932414683c8544a636b7b28 (127.13.223.65:39033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d1cb291cdd0444f5aa01d6cdcb32914f: Network error: Client connection negotiation failed: client connection to 127.13.223.65:39033: connect: Connection refused (error 111)
I20260928 14:46:50.579205 24460 catalog_manager.cc:4841] TS dd5cca0c08384ee19660ec87004d9507 (127.13.223.67:40351): tablet d1cb291cdd0444f5aa01d6cdcb32914f (table test-table [id=e27ebdeac0804a83be5a54723974bf18]) successfully deleted
I20260928 14:46:50.583829 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24804
I20260928 14:46:50.584115 24948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:46:50.583729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:41909 (local address 127.0.0.1:52666)
0928 14:46:50.583924 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0928 14:46:50.584001 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:41909: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 14:46:50.589529 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 24442
2026-09-28T14:46:50Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2636 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24978](info): set up 2 sockets
Sep 28 14:46:50 dist-test-slave-gkhx krb5kdc[24978](info): commencing operation
krb5kdc: starting...
W20260928 14:46:51.689489 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:46:51 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:46:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:46:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangeradmin_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangeradmin_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangerlookup_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangerlookup_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
I20260928 14:46:51.890218 14205 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktMPbyY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/postgres -l logfile start
2026-09-28 14:46:55.043 UTC [25008] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:46:55.043 UTC [25008] LOG: listening on IPv4 address "127.13.223.84", port 60689
2026-09-28 14:46:55.049 UTC [25008] LOG: listening on Unix socket "/tmp/.s.PGSQL.60689"
2026-09-28 14:46:55.074 UTC [25011] LOG: database system was shut down at 2026-09-28 14:46:53 UTC
2026-09-28 14:46:55.079 UTC [25008] LOG: database system is ready to accept connections
I20260928 14:46:56.089463 14205 mini_postgres.cc:96] Postgres bound to 60689
2026-09-28 14:46:56.097 UTC [25019] FATAL: database "slave" does not exist
127.13.223.84:60689 - accepting connections
I20260928 14:46:56.098825 14205 mini_ranger.cc:161] Starting Ranger...
I20260928 14:46:56.116921 14205 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:46:56.827690 14205 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:46:56.827798 14205 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin
2026-09-28 14:46:57,214 [I] DB FLAVOR :POSTGRES
2026-09-28 14:46:57,215 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:46:57,215 [I] Checking connection..
2026-09-28 14:46:57,215 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:46:57,498 [I] Checking connection passed.
2026-09-28 14:46:57,498 [I] --------- Verifying version history table ---------
2026-09-28 14:46:57,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:46:57,780 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:46:57,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:46:58,065 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:46:58,065 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:46:58,066 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:46:58.347 UTC [25083] WARNING: there is no transaction in progress
2026-09-28 14:46:58,355 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:46:58,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:46:58,620 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:46:58,620 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:46:58,620 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:46:58,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:46:59,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkhx.c.gcp-upstream.internal','N') ;"
2026-09-28 14:46:59,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:46:59,729 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:46:59,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:46:59,994 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:46:59,994 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:47:00,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:47:00,562 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:47:00,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:47:01.840 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:01.862 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:01.883 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:01.904 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:01.926 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:01.952 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.028 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.038 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.062 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.076 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.090 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.109 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.123 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.133 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.144 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:02.155 UTC [25192] WARNING: there is no transaction in progress
2026-09-28 14:47:03,151 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:47:03,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:47:03,405 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:47:03,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:47:03,656 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:47:03,656 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:47:03,923 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:47:03,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:47:04,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:47:04,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkhx.c.gcp-upstream.internal';"
2026-09-28 14:47:04,721 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:47:04,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:47:05,008 [I] DB_PATCHES have already been applied
I20260928 14:47:05.015082 14205 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktMPbyY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:47:05.015162 14205 mini_ranger.cc:193] Using host: 127.13.223.84
I20260928 14:47:05.019083 14205 mini_ranger.cc:227] Ranger admin URL: http://127.13.223.84:36549
Sep 28, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:47:05 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:47:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:47:06.183533 14205 mini_ranger.cc:160] Time spent starting Ranger: real 10.085s user 0.006s sys 0.005s
Sep 28, 2026 2:47:27 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:47:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 28, 2026 2:47:28 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:47:28 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:47:29 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:47:29.865594 14205 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangerkms_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/rangerkms_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.223.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:47:29 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606849, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:47:29.969637 14205 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:47:29.980386 14205 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:47:31.295367 14205 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:47:31.295485 14205 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms
I20260928 14:47:31.295599 14205 mini_ranger_kms.cc:227] Using postgres at 127.13.223.84:60689
2026-09-28 14:47:31,790 [I] DB FLAVOR :POSTGRES
2026-09-28 14:47:31,791 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:47:31,791 [I] Checking connection
2026-09-28 14:47:31,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:47:32,061 [I] connection success
2026-09-28 14:47:32,061 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:47:32,061 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:47:32,062 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:47:32,359 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:47:32,360 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:47:32,360 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:47:32,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:60689/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:47:32,668 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:47:33.303630 14205 mini_ranger_kms.cc:310] Created kms service
I20260928 14:47:33.526672 14205 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:47:33.578503 14205 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:47:33 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606849, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:34.064424 14205 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:47:34.064795 14205 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:47:34.064874 14205 mini_ranger_kms.cc:240] Using host: 127.13.223.84
I20260928 14:47:34.068740 14205 mini_ranger_kms.cc:276] Ranger KMS PID: 25374
I20260928 14:47:34.068897 14205 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.223.84:45945
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:47:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:47:36.459694 14205 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.490s user 0.000s sys 0.259s
I20260928 14:47:36.459894 14205 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:47:36.460120 14205 mini_ranger_kms.cc:401] 127.13.223.84:45945/kms/v1/keys
Sep 28 14:47:36 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606856, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:47:37 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606856, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
Sep 28 14:47:37 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606856, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
Sep 28 14:47:40 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:47:40.066852 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37023
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:37023
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:45945/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:40.185094 25425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:40.185391 25425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:40.185518 25425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:40.189704 25425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:47:40.189781 25425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:40.189801 25425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:40.189818 25425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:47:40.189834 25425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:47:40.195760 25425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.223.84:45945/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:37023
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:47:40.197012 25425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:40.198129 25425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:40.208391 25430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.208391 25431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:40.208591 25433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:40.208824 25425 server_base.cc:1034] running on GCE node
I20260928 14:47:40.209506 25425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:40.210867 25425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:40.212121 25425 hybrid_clock.cc:648] HybridClock initialized: now 1790606860212082 us; error 48 us; skew 500 ppm
Sep 28 14:47:40 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:40.216351 25425 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:47:40.217424 25425 webserver.cc:460] Webserver started at http://127.13.223.126:42317/ using document root <none> and password file <none>
I20260928 14:47:40.218051 25425 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:40.218113 25425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:40.218403 25425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:47:40 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:40.274464 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "efcb15736f964d1e99157252dd0bc4ae"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-gkhx"
server_key: "28f33b0d0e43a5c7fa9d3dbc3f31b3c0"
server_key_iv: "2054c63bd3966a8150853ac3ec64ae8f"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:40.275418 25425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "efcb15736f964d1e99157252dd0bc4ae"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-gkhx"
server_key: "28f33b0d0e43a5c7fa9d3dbc3f31b3c0"
server_key_iv: "2054c63bd3966a8150853ac3ec64ae8f"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:40.280555 25425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:47:40.301204 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:40.303187 25425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:47:40.303385 25425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "efcb15736f964d1e99157252dd0bc4ae"
format_stamp: "Formatted at 2026-09-28 14:47:40 on dist-test-slave-gkhx"
server_key: "28f33b0d0e43a5c7fa9d3dbc3f31b3c0"
server_key_iv: "2054c63bd3966a8150853ac3ec64ae8f"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:40.303601 25425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:40.319079 25425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:40.322171 25425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:40.322435 25425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:40.330514 25425 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:37023
I20260928 14:47:40.330678 25492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:37023 every 8 connection(s)
I20260928 14:47:40.331677 25425 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:47:40.334854 25493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:40.335724 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 25425
I20260928 14:47:40.335886 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:47:40.336205 14205 external_mini_cluster.cc:1428] Setting key 02d9112724698fedd0b71796151b99ea
I20260928 14:47:40.342519 25493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: Bootstrap starting.
I20260928 14:47:40.345273 25493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:40.346103 25493 log.cc:824] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:47:40 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606860, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:47:40.350378 25493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: No bootstrap required, opened a new log
I20260928 14:47:40.354696 25496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:40.337663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.13.223.126:37023)
0928 14:47:40.338139 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:40.338161 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:40.338207 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:47:40.339100 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:40.339126 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:40.339596 (+ 470us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:40.339997 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:40.340906 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.342450 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.343787 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:40.344496 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:40.344672 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:40.347706 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:40.347713 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:40.347731 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:40.347763 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:40.350819 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.351395 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.351403 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.351411 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.351582 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:40.351947 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:40.351950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:40.351952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:40.352575 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:40.352802 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:40.353161 (+ 359us) server_negotiation.cc:299] Negotiation successful
0928 14:47:40.353495 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":138,"threads_started":1}
I20260928 14:47:40.355093 25493 raft_consensus.cc:348] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } }
I20260928 14:47:40.355405 25493 raft_consensus.cc:374] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:40.355502 25493 raft_consensus.cc:729] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efcb15736f964d1e99157252dd0bc4ae, State: Initialized, Role: FOLLOWER
I20260928 14:47:40.356063 25493 consensus_queue.cc:260] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } }
I20260928 14:47:40.356256 25493 raft_consensus.cc:388] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:47:40.356357 25493 raft_consensus.cc:482] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:47:40.356498 25493 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:40.357790 25493 raft_consensus.cc:504] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } }
I20260928 14:47:40.358207 25493 leader_election.cc:302] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efcb15736f964d1e99157252dd0bc4ae; no voters:
I20260928 14:47:40.358683 25493 leader_election.cc:288] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:47:40.358800 25498 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:40.359138 25498 raft_consensus.cc:686] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [term 1 LEADER]: Becoming Leader. State: Replica: efcb15736f964d1e99157252dd0bc4ae, State: Running, Role: LEADER
I20260928 14:47:40.359508 25498 consensus_queue.cc:237] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } }
I20260928 14:47:40.360127 25493 sys_catalog.cc:564] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:47:40.361445 25500 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader efcb15736f964d1e99157252dd0bc4ae. Latest consensus state: current_term: 1 leader_uuid: "efcb15736f964d1e99157252dd0bc4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } } }
I20260928 14:47:40.361609 25500 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:40.362000 25499 sys_catalog.cc:455] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efcb15736f964d1e99157252dd0bc4ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efcb15736f964d1e99157252dd0bc4ae" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37023 } } }
I20260928 14:47:40.362071 25507 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:47:40.362747 25499 sys_catalog.cc:458] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae [sys.catalog]: This master's current role is: LEADER
I20260928 14:47:40.363289 25493 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:47:40.368880 25507 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:47:40.377153 25507 catalog_manager.cc:1347] Generated new cluster ID: 17f9fdd2c61c4856a54ef7440900c755
I20260928 14:47:40.377244 25507 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:47:40.399863 25507 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:47:40.401350 25507 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:47:40.415016 25507 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: Generated new TSK 0
I20260928 14:47:40.416036 25507 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:47:41.139638 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:45945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:41.304037 25539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:41.304549 25539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:41.304636 25539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:41.311708 25539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:41.311889 25539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:41.312006 25539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:47:41.321204 25539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.223.84:45945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:47:41.322886 25539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:41.324389 25539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:41.340025 25546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:41.340412 25547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:41.341904 25539 server_base.cc:1034] running on GCE node
W20260928 14:47:41.342304 25549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:41.342941 25539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:41.343961 25539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:41.347332 25539 hybrid_clock.cc:648] HybridClock initialized: now 1790606861347296 us; error 51 us; skew 500 ppm
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:41.351557 25539 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:47:41.352624 25539 webserver.cc:460] Webserver started at http://127.13.223.65:42339/ using document root <none> and password file <none>
I20260928 14:47:41.353272 25539 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:41.353374 25539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:41.353636 25539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:41.382099 25539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "bdd36109bf72440dabddc410c563719c"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "a3d5427fa0751a6e773131d4de7d6063"
server_key_iv: "699ec5387e7496645ae1239cb810e73b"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.383503 25539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "bdd36109bf72440dabddc410c563719c"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "a3d5427fa0751a6e773131d4de7d6063"
server_key_iv: "699ec5387e7496645ae1239cb810e73b"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.389477 25539 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:47:41.408638 25558 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.411788 25539 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 14:47:41.411954 25539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "bdd36109bf72440dabddc410c563719c"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "a3d5427fa0751a6e773131d4de7d6063"
server_key_iv: "699ec5387e7496645ae1239cb810e73b"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.412122 25539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:41.449350 25539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:41.452757 25539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:41.452986 25539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:41.453694 25539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:41.454725 25539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:41.454794 25539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.454897 25539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:41.454934 25539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.466156 25539 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:42549
I20260928 14:47:41.466177 25671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:42549 every 8 connection(s)
I20260928 14:47:41.467403 25539 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:47:41.476539 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 25539
I20260928 14:47:41.476694 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:47:41.477087 14205 external_mini_cluster.cc:1428] Setting key 89ff68558a5f30445d1b1bfef4574a49
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:47:41.484097 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.469717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:56881 (local address 127.13.223.126:37023)
0928 14:47:41.469984 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.469990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.470955 (+ 965us) server_negotiation.cc:407] Connection header received
0928 14:47:41.472096 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.472100 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.472170 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.472297 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.473536 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.474291 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.475635 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.475890 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.476020 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.479178 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.479182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.479185 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.479212 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.481335 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.482163 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.482169 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.482173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.482279 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.482791 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.482797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.482800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.483017 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.483086 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.483740 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.483887 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":81,"threads_started":1}
I20260928 14:47:41.485090 25675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.470230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37023 (local address 127.13.223.65:56881)
0928 14:47:41.470777 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.470836 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.471844 (+ 1008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.472518 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.472526 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.472826 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:41.473356 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.473370 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.474440 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.474451 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.475489 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.475499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.476075 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.476079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.476347 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.477133 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:41.477157 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:41.478965 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:41.481539 (+ 2574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.481547 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.481574 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.482022 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.482395 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.482401 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.482403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.482668 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.483145 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:41.483150 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:41.483469 (+ 319us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.483685 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.484053 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":160,"threads_started":1}
I20260928 14:47:41.490681 25672 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37023
I20260928 14:47:41.490983 25672 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:41.491590 25672 heartbeater.cc:511] Master 127.13.223.126:37023 requested a full tablet report, sending...
I20260928 14:47:41.493587 25457 ts_manager.cc:194] Registered new tserver with Master: bdd36109bf72440dabddc410c563719c (127.13.223.65:42549)
I20260928 14:47:41.495373 25457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:56881
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:47:41.522658 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:45945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:47:41.665396 25678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:41.665809 25678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:41.665887 25678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:41.672164 25678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:41.672279 25678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:41.672403 25678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:47:41.678879 25678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.223.84:45945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:47:41.680485 25678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:41.682155 25678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:41.696251 25685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:41.696374 25687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:41.696658 25684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:41.698652 25678 server_base.cc:1034] running on GCE node
I20260928 14:47:41.699277 25678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:41.700249 25678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:41.701505 25678 hybrid_clock.cc:648] HybridClock initialized: now 1790606861701479 us; error 51 us; skew 500 ppm
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:41.705813 25678 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:47:41.707264 25678 webserver.cc:460] Webserver started at http://127.13.223.66:38445/ using document root <none> and password file <none>
I20260928 14:47:41.708130 25678 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:41.708294 25678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:41.708647 25678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:41.726956 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "68f29e36bdfa43f5b245bbcfb88193a9"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "79a9ebd5209d0563e82acb7721ea5073"
server_key_iv: "13829e3f23190f7fb075b325b70526ab"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.728044 25678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "68f29e36bdfa43f5b245bbcfb88193a9"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "79a9ebd5209d0563e82acb7721ea5073"
server_key_iv: "13829e3f23190f7fb075b325b70526ab"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.734251 25678 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260928 14:47:41.753517 25694 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.755574 25678 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260928 14:47:41.755753 25678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "68f29e36bdfa43f5b245bbcfb88193a9"
format_stamp: "Formatted at 2026-09-28 14:47:41 on dist-test-slave-gkhx"
server_key: "79a9ebd5209d0563e82acb7721ea5073"
server_key_iv: "13829e3f23190f7fb075b325b70526ab"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:41.755919 25678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:41.778877 25678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:41.783224 25678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:41.783542 25678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:41.784699 25678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:41.785948 25678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:41.786063 25678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.786176 25678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:41.786249 25678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:41.800967 25678 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37697
I20260928 14:47:41.801088 25808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37697 every 8 connection(s)
I20260928 14:47:41.802245 25678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:47:41.803673 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 25678
I20260928 14:47:41.803803 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:47:41.804136 14205 external_mini_cluster.cc:1428] Setting key 5383c1ff0ab72f49c200e15d0bc07a59
Sep 28 14:47:41 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:47:41.823316 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.804525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:54339 (local address 127.13.223.126:37023)
0928 14:47:41.804678 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:41.804683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:41.808530 (+ 3847us) server_negotiation.cc:407] Connection header received
0928 14:47:41.809902 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:41.809906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:41.809980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:41.810078 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:41.811878 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.812722 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.814504 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.814910 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.815031 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.818339 (+ 3308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:41.818343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:41.818346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:41.818378 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:41.820332 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.821043 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.821047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.821049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.821160 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:41.821693 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:41.821697 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:41.821699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:41.821962 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:41.822093 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:41.822909 (+ 816us) server_negotiation.cc:299] Negotiation successful
0928 14:47:41.823138 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 14:47:41.825464 25811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:41.805105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37023 (local address 127.13.223.66:54339)
0928 14:47:41.808343 (+ 3238us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:41.808418 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:41.809611 (+ 1193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:41.810432 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:41.810443 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:41.810858 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:41.811667 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.811690 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.812910 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.812914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.814344 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:41.814357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:41.815233 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:41.815237 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:41.815452 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:41.816210 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:41.816233 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:41.818107 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:41.820472 (+ 2365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.820481 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.820502 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.820897 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.821348 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:41.821351 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:41.821353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:41.821574 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:41.822155 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:41.822163 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:41.822592 (+ 429us) client_negotiation.cc:769] Sending connection context
0928 14:47:41.823826 (+ 1234us) client_negotiation.cc:240] Negotiation successful
0928 14:47:41.824275 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2958,"thread_start_us":146,"threads_started":1}
I20260928 14:47:41.827041 25809 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37023
I20260928 14:47:41.827353 25809 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:41.828088 25809 heartbeater.cc:511] Master 127.13.223.126:37023 requested a full tablet report, sending...
I20260928 14:47:41.829598 25457 ts_manager.cc:194] Registered new tserver with Master: 68f29e36bdfa43f5b245bbcfb88193a9 (127.13.223.66:37697)
I20260928 14:47:41.830519 25457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:54339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:47:41.863222 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:45945/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:41967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:47:42 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
W20260928 14:47:42.056705 25814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:47:42.057081 25814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:47:42.057163 25814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:47:42.063678 25814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:47:42.063822 25814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:47:42.063977 25814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:47:42.072086 25814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.223.84:45945/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:47:42.074028 25814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:47:42.075567 25814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:47:42.109719 25819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.109787 25822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:47:42.109668 25820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:47:42.116344 25814 server_base.cc:1034] running on GCE node
I20260928 14:47:42.117048 25814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:47:42.118019 25814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:47:42.149710 25814 hybrid_clock.cc:648] HybridClock initialized: now 1790606862149626 us; error 238 us; skew 500 ppm
Sep 28 14:47:42 dist-test-slave-gkhx krb5kdc[24978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:47:42.154601 25814 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:47:42.156073 25814 webserver.cc:460] Webserver started at http://127.13.223.67:35183/ using document root <none> and password file <none>
I20260928 14:47:42.156922 25814 fs_manager.cc:359] Metadata directory not provided
I20260928 14:47:42.157091 25814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:47:42.157413 25814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:47:42 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:42.188438 25814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "cf48b8d0a6754ab59351e1adf2b22320"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-gkhx"
server_key: "b4edc4475968ae2ec28481c454696f31"
server_key_iv: "b9d35dce429d0434370e5584d58df550"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:42.189509 25814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "cf48b8d0a6754ab59351e1adf2b22320"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-gkhx"
server_key: "b4edc4475968ae2ec28481c454696f31"
server_key_iv: "b9d35dce429d0434370e5584d58df550"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:42.208184 25814 fs_manager.cc:691] Time spent creating directory manager: real 0.018s user 0.001s sys 0.004s
I20260928 14:47:42.233738 25829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.235778 25814 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260928 14:47:42.235946 25814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "cf48b8d0a6754ab59351e1adf2b22320"
format_stamp: "Formatted at 2026-09-28 14:47:42 on dist-test-slave-gkhx"
server_key: "b4edc4475968ae2ec28481c454696f31"
server_key_iv: "b9d35dce429d0434370e5584d58df550"
server_key_version: "kuduclusterkey@0"
I20260928 14:47:42.236111 25814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:47:42.265237 25814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:47:42.269624 25814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:47:42.269934 25814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:47:42.270835 25814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:47:42.272055 25814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:47:42.272168 25814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.272272 25814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:47:42.272346 25814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:47:42.286818 25814 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:33717
I20260928 14:47:42.288998 25942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:33717 every 8 connection(s)
I20260928 14:47:42.289631 25814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:47:42.294863 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 25814
I20260928 14:47:42.295012 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:47:42.295341 14205 external_mini_cluster.cc:1428] Setting key 9ec7ee6d73428404e8aeabee7e43451b
Sep 28 14:47:42 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606862, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:47:42.311414 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.295318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:47427 (local address 127.13.223.126:37023)
0928 14:47:42.295480 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.295485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.296012 (+ 527us) server_negotiation.cc:407] Connection header received
0928 14:47:42.297385 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.297388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.297446 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.297514 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.299404 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.300215 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.302209 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.302613 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.302704 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.306525 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.306530 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.306534 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.306594 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.308455 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.309220 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.309225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.309229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.309329 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.309835 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.309840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.309842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.310094 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.310171 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.311070 (+ 899us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.311229 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:47:42.313601 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.293452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37023 (local address 127.13.223.67:47427)
0928 14:47:42.295839 (+ 2387us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.295910 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.297131 (+ 1221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.298000 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.298011 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.298402 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:47:42.299208 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.299230 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.300401 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.300404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.302035 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.302050 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.302969 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.302974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.303228 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.304281 (+ 1053us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:47:42.304307 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:47:42.306222 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:47:42.308634 (+ 2412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.308644 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.308669 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.309106 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.309453 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:47:42.309456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:47:42.309471 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:47:42.309711 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:47:42.310265 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:47:42.310272 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:47:42.310731 (+ 459us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.311974 (+ 1243us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.312401 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2110,"thread_start_us":179,"threads_started":1}
I20260928 14:47:42.315883 25943 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37023
I20260928 14:47:42.316182 25943 heartbeater.cc:464] Registering TS with master...
I20260928 14:47:42.316915 25943 heartbeater.cc:511] Master 127.13.223.126:37023 requested a full tablet report, sending...
I20260928 14:47:42.318382 25457 ts_manager.cc:194] Registered new tserver with Master: cf48b8d0a6754ab59351e1adf2b22320 (127.13.223.67:33717)
I20260928 14:47:42.319271 25457 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:47427
I20260928 14:47:42.321475 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:47:42.357839 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.327245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40668 (local address 127.13.223.126:37023)
0928 14:47:42.327392 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.327398 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.338870 (+ 11472us) server_negotiation.cc:407] Connection header received
0928 14:47:42.339412 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.339416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.339478 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.339545 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:47:42.340761 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.341718 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.343541 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.343920 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.344028 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.346273 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:47:42.346278 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:47:42.346281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:47:42.346322 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:47:42.352992 (+ 6670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.355769 (+ 2777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.355775 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.355777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.355924 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:47:42.356553 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:47:42.356560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:47:42.356563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:47:42.356895 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:47:42.357094 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.357504 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.357665 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:47:42.362407 25457 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:47:42.368821 25457 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:47:42.397655 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.388176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37697 (local address 127.0.0.1:49942)
0928 14:47:42.390742 (+ 2566us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.390760 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.390879 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.391594 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.391597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.391618 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:42.392130 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.392136 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.393681 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.393684 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.395376 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.395394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.396956 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.396960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.397125 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.397169 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.397297 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.397467 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2483,"mutex_wait_us":341,"thread_start_us":162,"threads_started":1}
I20260928 14:47:42.398342 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.388370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49942 (local address 127.13.223.66:37697)
0928 14:47:42.388945 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.388954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.390992 (+ 2038us) server_negotiation.cc:407] Connection header received
0928 14:47:42.391088 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.391093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.391254 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.391417 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:42.392263 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.393511 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.395568 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.396569 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.396693 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.396886 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.398056 (+ 1170us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.398186 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":107,"threads_started":1}
I20260928 14:47:42.399448 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.384060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:33717 (local address 127.0.0.1:35508)
0928 14:47:42.389272 (+ 5212us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.389347 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.389509 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.391570 (+ 2061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.391578 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.391605 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:42.391846 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.391855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.396074 (+ 4219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.396078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.397322 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.397331 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.398942 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.398945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.399069 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.399098 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.399178 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.399302 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4969,"mutex_wait_us":1,"thread_start_us":151,"threads_started":1}
I20260928 14:47:42.400278 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.384177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35508 (local address 127.13.223.67:33717)
0928 14:47:42.390797 (+ 6620us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.390813 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.390846 (+ 33us) server_negotiation.cc:407] Connection header received
0928 14:47:42.390942 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.390952 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.391239 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.391417 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:42.394667 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.395912 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.397501 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.398491 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.398696 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.398880 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.399981 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.400110 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6488,"thread_start_us":90,"threads_started":1}
I20260928 14:47:42.402691 25740 tablet_service.cc:1467] Processing CreateTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 (DEFAULT_TABLE table=test-table [id=404a939b00024522ac21df557678e254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:42.404173 25740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b981032565a4ef1865ce02c3c4ff872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:42.405961 25877 tablet_service.cc:1467] Processing CreateTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 (DEFAULT_TABLE table=test-table [id=404a939b00024522ac21df557678e254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:42.407609 25877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b981032565a4ef1865ce02c3c4ff872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:42.420642 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.390393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42549 (local address 127.0.0.1:54862)
0928 14:47:42.391862 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.391878 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.391963 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.415571 (+ 23608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.415575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.415605 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:42.415734 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.415743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.417474 (+ 1731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.417478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.418685 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.418703 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.420132 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.420136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.420243 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.420272 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.420356 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.420453 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1406,"thread_start_us":162,"threads_started":1}
I20260928 14:47:42.421717 25962 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Bootstrap starting.
I20260928 14:47:42.422251 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.390914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.13.223.65:42549)
0928 14:47:42.414778 (+ 23864us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.414789 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.414827 (+ 38us) server_negotiation.cc:407] Connection header received
0928 14:47:42.414935 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.414941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.415182 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.415371 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:42.415984 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.417304 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.418866 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.419773 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.419887 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.420051 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.421965 (+ 1914us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.422104 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23716,"thread_start_us":91,"threads_started":1}
I20260928 14:47:42.425298 25962 tablet_bootstrap.cc:654] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:42.425984 25963 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Bootstrap starting.
I20260928 14:47:42.429260 25963 tablet_bootstrap.cc:654] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:42.428857 25588 tablet_service.cc:1467] Processing CreateTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 (DEFAULT_TABLE table=test-table [id=404a939b00024522ac21df557678e254]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:47:42.430302 25588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b981032565a4ef1865ce02c3c4ff872. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:47:42.430325 25963 log.cc:824] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:42.431859 25962 log.cc:824] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:42.434270 25963 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: No bootstrap required, opened a new log
I20260928 14:47:42.434512 25963 ts_tablet_manager.cc:1397] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 14:47:42.438736 25963 raft_consensus.cc:348] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.439033 25963 raft_consensus.cc:374] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:42.439102 25963 raft_consensus.cc:729] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf48b8d0a6754ab59351e1adf2b22320, State: Initialized, Role: FOLLOWER
I20260928 14:47:42.439687 25963 consensus_queue.cc:260] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.440831 25963 ts_tablet_manager.cc:1428] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:47:42.441299 25943 heartbeater.cc:503] Master 127.13.223.126:37023 was elected leader, sending a full tablet report...
I20260928 14:47:42.441635 25966 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Bootstrap starting.
I20260928 14:47:42.441826 25962 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: No bootstrap required, opened a new log
I20260928 14:47:42.442019 25962 ts_tablet_manager.cc:1397] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20260928 14:47:42.444396 25966 tablet_bootstrap.cc:654] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Neither blocks nor log segments found. Creating new log.
I20260928 14:47:42.445358 25966 log.cc:824] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:47:42.446852 25962 raft_consensus.cc:348] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.447185 25962 raft_consensus.cc:374] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:42.447278 25962 raft_consensus.cc:729] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68f29e36bdfa43f5b245bbcfb88193a9, State: Initialized, Role: FOLLOWER
I20260928 14:47:42.447903 25962 consensus_queue.cc:260] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.449265 25962 ts_tablet_manager.cc:1428] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 14:47:42.449438 25809 heartbeater.cc:503] Master 127.13.223.126:37023 was elected leader, sending a full tablet report...
I20260928 14:47:42.451088 25966 tablet_bootstrap.cc:492] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: No bootstrap required, opened a new log
I20260928 14:47:42.451406 25966 ts_tablet_manager.cc:1397] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260928 14:47:42.458726 25966 raft_consensus.cc:348] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.459158 25966 raft_consensus.cc:374] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:47:42.459246 25966 raft_consensus.cc:729] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdd36109bf72440dabddc410c563719c, State: Initialized, Role: FOLLOWER
I20260928 14:47:42.459810 25966 consensus_queue.cc:260] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.461589 25966 ts_tablet_manager.cc:1428] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260928 14:47:42.462146 25672 heartbeater.cc:503] Master 127.13.223.126:37023 was elected leader, sending a full tablet report...
W20260928 14:47:42.469908 25673 tablet.cc:2369] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:47:42 dist-test-slave-gkhx krb5kdc[24978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606861, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:47:42.508414 25967 raft_consensus.cc:482] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:47:42.508752 25967 raft_consensus.cc:504] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.510516 25967 leader_election.cc:288] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68f29e36bdfa43f5b245bbcfb88193a9 (127.13.223.66:37697), bdd36109bf72440dabddc410c563719c (127.13.223.65:42549)
I20260928 14:47:42.516264 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.511021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37697 (local address 127.13.223.67:57409)
0928 14:47:42.511159 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.511179 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.511291 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.511732 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.511737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.511759 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:42.511909 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.511919 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.513182 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.513186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.514503 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.514518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.515743 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.515747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.515880 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.515908 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.515999 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.516095 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 14:47:42.517205 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.511127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:57409 (local address 127.13.223.66:37697)
0928 14:47:42.511272 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.511278 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.511298 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:47:42.511409 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.511412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.511473 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.511582 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:42.512081 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.512908 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.514761 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.515462 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.515606 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.515672 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.516950 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.517063 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:47:42.518013 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b981032565a4ef1865ce02c3c4ff872" candidate_uuid: "cf48b8d0a6754ab59351e1adf2b22320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" is_pre_election: true
I20260928 14:47:42.518267 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.511824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:43063 (local address 127.13.223.65:42549)
0928 14:47:42.511970 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.511975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.512430 (+ 455us) server_negotiation.cc:407] Connection header received
0928 14:47:42.512658 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.512662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.512729 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.512784 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:47:42.513243 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.514127 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.516612 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.517327 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.517450 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.517520 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.518017 (+ 497us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.518121 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:47:42.518388 25757 raft_consensus.cc:2454] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48b8d0a6754ab59351e1adf2b22320 in term 0.
I20260928 14:47:42.518787 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.511372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42549 (local address 127.13.223.67:43063)
0928 14:47:42.512317 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.512336 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:47:42.512514 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:47:42.512953 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:47:42.512956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:47:42.512974 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:47:42.513104 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.513110 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.515091 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.515095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.516441 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:47:42.516454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.517701 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.517706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:47:42.517831 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.517862 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:47:42.518483 (+ 621us) client_negotiation.cc:240] Negotiation successful
0928 14:47:42.518602 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":139,"threads_started":1}
I20260928 14:47:42.519447 25830 leader_election.cc:302] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68f29e36bdfa43f5b245bbcfb88193a9, cf48b8d0a6754ab59351e1adf2b22320; no voters:
I20260928 14:47:42.519551 25608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b981032565a4ef1865ce02c3c4ff872" candidate_uuid: "cf48b8d0a6754ab59351e1adf2b22320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd36109bf72440dabddc410c563719c" is_pre_election: true
I20260928 14:47:42.519829 25967 raft_consensus.cc:2781] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:47:42.519922 25608 raft_consensus.cc:2454] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf48b8d0a6754ab59351e1adf2b22320 in term 0.
I20260928 14:47:42.519992 25967 raft_consensus.cc:482] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:47:42.520045 25967 raft_consensus.cc:3037] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:42.521624 25967 raft_consensus.cc:504] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.522112 25967 leader_election.cc:288] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [CANDIDATE]: Term 1 election: Requested vote from peers 68f29e36bdfa43f5b245bbcfb88193a9 (127.13.223.66:37697), bdd36109bf72440dabddc410c563719c (127.13.223.65:42549)
I20260928 14:47:42.522811 25608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b981032565a4ef1865ce02c3c4ff872" candidate_uuid: "cf48b8d0a6754ab59351e1adf2b22320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bdd36109bf72440dabddc410c563719c"
I20260928 14:47:42.522961 25757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b981032565a4ef1865ce02c3c4ff872" candidate_uuid: "cf48b8d0a6754ab59351e1adf2b22320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68f29e36bdfa43f5b245bbcfb88193a9"
I20260928 14:47:42.522980 25608 raft_consensus.cc:3037] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:42.523108 25757 raft_consensus.cc:3037] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:47:42.524212 25608 raft_consensus.cc:2454] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf48b8d0a6754ab59351e1adf2b22320 in term 1.
I20260928 14:47:42.524521 25757 raft_consensus.cc:2454] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf48b8d0a6754ab59351e1adf2b22320 in term 1.
I20260928 14:47:42.524643 25833 leader_election.cc:302] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bdd36109bf72440dabddc410c563719c, cf48b8d0a6754ab59351e1adf2b22320; no voters:
I20260928 14:47:42.525013 25967 raft_consensus.cc:2781] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:47:42.525290 25967 raft_consensus.cc:686] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 1 LEADER]: Becoming Leader. State: Replica: cf48b8d0a6754ab59351e1adf2b22320, State: Running, Role: LEADER
I20260928 14:47:42.525694 25967 consensus_queue.cc:237] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } }
I20260928 14:47:42.529817 25457 catalog_manager.cc:5495] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 reported cstate change: term changed from 0 to 1, leader changed from <none> to cf48b8d0a6754ab59351e1adf2b22320 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "cf48b8d0a6754ab59351e1adf2b22320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf48b8d0a6754ab59351e1adf2b22320" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 33717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 } health_report { overall_health: UNKNOWN } } }
W20260928 14:47:42.541235 25944 tablet.cc:2369] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:47:42.554280 25810 tablet.cc:2369] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:47:42.565909 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.559831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35514 (local address 127.13.223.67:33717)
0928 14:47:42.560014 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:47:42.560021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:47:42.560042 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:47:42.560152 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:47:42.560156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:47:42.560237 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:47:42.560400 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:47:42.560740 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.561697 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.563231 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:47:42.563669 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:47:42.563782 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:47:42.564290 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:47:42.564967 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:47:42.565104 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:47:42.565664 (+ 560us) server_negotiation.cc:299] Negotiation successful
0928 14:47:42.565742 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:47:42.580869 25608 raft_consensus.cc:1261] T 2b981032565a4ef1865ce02c3c4ff872 P bdd36109bf72440dabddc410c563719c [term 1 FOLLOWER]: Refusing update from remote peer cf48b8d0a6754ab59351e1adf2b22320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:42.581933 25967 consensus_queue.cc:1035] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bdd36109bf72440dabddc410c563719c" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:42.594597 25757 raft_consensus.cc:1261] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 1 FOLLOWER]: Refusing update from remote peer cf48b8d0a6754ab59351e1adf2b22320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:47:42.595600 25967 consensus_queue.cc:1035] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68f29e36bdfa43f5b245bbcfb88193a9" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:47:42.613610 25977 mvcc.cc:205] Tried to move back new op lower bound from 7334325709122064384 to 7334325708908806144. Current Snapshot: MvccSnapshot[applied={T|T < 7334325709122064384}]
I20260928 14:47:42.631825 25457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:42.632072 25457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:47:42.632030 25980 mvcc.cc:205] Tried to move back new op lower bound from 7334325709122064384 to 7334325708908806144. Current Snapshot: MvccSnapshot[applied={T|T < 7334325709122064384 or (T in {7334325709122064384})}]
I20260928 14:47:42.616616 25978 mvcc.cc:205] Tried to move back new op lower bound from 7334325709122064384 to 7334325708908806144. Current Snapshot: MvccSnapshot[applied={T|T < 7334325709122064384}]
I20260928 14:47:42.635509 25457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P efcb15736f964d1e99157252dd0bc4ae: Sending DeleteTablet for 3 replicas of tablet 2b981032565a4ef1865ce02c3c4ff872
I20260928 14:47:42.636550 25877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:42 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:35508
I20260928 14:47:42.636842 25740 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:42 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49942
I20260928 14:47:42.638039 25588 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b981032565a4ef1865ce02c3c4ff872 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:47:42 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:54862
I20260928 14:47:42.639915 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 25539
I20260928 14:47:42.650882 25987 tablet_replica.cc:330] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: stopping tablet replica
I20260928 14:47:42.651460 25987 raft_consensus.cc:2229] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:47:42.655587 25987 raft_consensus.cc:2258] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:42.656179 25986 tablet_replica.cc:330] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: stopping tablet replica
I20260928 14:47:42.656615 25986 raft_consensus.cc:2229] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:47:42.657137 25986 raft_consensus.cc:2258] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:47:42.659179 25987 ts_tablet_manager.cc:1905] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:42.662683 25987 ts_tablet_manager.cc:1918] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:47:42.662847 25987 log.cc:1196] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/2b981032565a4ef1865ce02c3c4ff872
I20260928 14:47:42.663255 25987 ts_tablet_manager.cc:1939] T 2b981032565a4ef1865ce02c3c4ff872 P 68f29e36bdfa43f5b245bbcfb88193a9: Deleting consensus metadata
W20260928 14:47:42.663496 25444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:42549 (error 108)
I20260928 14:47:42.664810 25441 catalog_manager.cc:4841] TS 68f29e36bdfa43f5b245bbcfb88193a9 (127.13.223.66:37697): tablet 2b981032565a4ef1865ce02c3c4ff872 (table test-table [id=404a939b00024522ac21df557678e254]) successfully deleted
I20260928 14:47:42.665586 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 25678
I20260928 14:47:42.671135 25986 ts_tablet_manager.cc:1905] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:47:42.674700 25986 ts_tablet_manager.cc:1918] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:47:42.674932 25986 log.cc:1196] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/2b981032565a4ef1865ce02c3c4ff872
I20260928 14:47:42.675449 25986 ts_tablet_manager.cc:1939] T 2b981032565a4ef1865ce02c3c4ff872 P cf48b8d0a6754ab59351e1adf2b22320: Deleting consensus metadata
I20260928 14:47:42.675752 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:47:42.675350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:42549 (local address 127.0.0.1:54868)
0928 14:47:42.675531 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0928 14:47:42.675636 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 14:47:42.675966 25444 catalog_manager.cc:4584] TS bdd36109bf72440dabddc410c563719c (127.13.223.65:42549): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b981032565a4ef1865ce02c3c4ff872: Network error: Client connection negotiation failed: client connection to 127.13.223.65:42549: connect: Connection refused (error 111)
I20260928 14:47:42.677080 25444 catalog_manager.cc:4841] TS cf48b8d0a6754ab59351e1adf2b22320 (127.13.223.67:33717): tablet 2b981032565a4ef1865ce02c3c4ff872 (table test-table [id=404a939b00024522ac21df557678e254]) successfully deleted
I20260928 14:47:42.682817 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 25814
I20260928 14:47:42.691916 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 25425
I20260928 14:47:42.700798 14205 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:47:43.047526 14205 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:47:43.047767 14205 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:47:43.400137 14205 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:47:43.400 UTC [25008] LOG: received smart shutdown request
2026-09-28 14:47:43.404 UTC [25008] LOG: background worker "logical replication launcher" (PID 25017) exited with exit code 1
2026-09-28 14:47:43.405 UTC [25012] LOG: shutting down
2026-09-28 14:47:43.635 UTC [25008] LOG: database system is shut down
2026-09-28T14:47:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (53167 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:47:43 dist-test-slave-gkhx krb5kdc[25998](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:47:43 dist-test-slave-gkhx krb5kdc[25998](info): set up 2 sockets
Sep 28 14:47:43 dist-test-slave-gkhx krb5kdc[25998](info): commencing operation
krb5kdc: starting...
W20260928 14:47:44.872839 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:47:44 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:47:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:47:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangeradmin_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangeradmin_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangerlookup_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangerlookup_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
I20260928 14:47:45.079468 14205 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-tasktMPbyY/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/postgres -l logfile start
2026-09-28 14:47:48.067 UTC [26028] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 14:47:48.067 UTC [26028] LOG: listening on IPv4 address "127.13.223.84", port 38395
2026-09-28 14:47:48.072 UTC [26028] LOG: listening on Unix socket "/tmp/.s.PGSQL.38395"
2026-09-28 14:47:48.095 UTC [26031] LOG: database system was shut down at 2026-09-28 14:47:46 UTC
2026-09-28 14:47:48.099 UTC [26028] LOG: database system is ready to accept connections
I20260928 14:47:49.080442 14205 mini_postgres.cc:96] Postgres bound to 38395
2026-09-28 14:47:49.087 UTC [26039] FATAL: database "slave" does not exist
127.13.223.84:38395 - accepting connections
I20260928 14:47:49.087919 14205 mini_ranger.cc:161] Starting Ranger...
I20260928 14:47:49.105278 14205 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 14:47:49.747853 14205 mini_ranger.cc:87] Created ranger Postgres database
I20260928 14:47:49.747943 14205 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin
2026-09-28 14:47:50,032 [I] DB FLAVOR :POSTGRES
2026-09-28 14:47:50,033 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:47:50,033 [I] Checking connection..
2026-09-28 14:47:50,033 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 14:47:50,308 [I] Checking connection passed.
2026-09-28 14:47:50,309 [I] --------- Verifying version history table ---------
2026-09-28 14:47:50,309 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:47:50,567 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:47:50,568 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:47:50,857 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 14:47:50,858 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 14:47:50,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 14:47:51.124 UTC [26103] WARNING: there is no transaction in progress
2026-09-28 14:47:51,131 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 14:47:51,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 14:47:51,409 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 14:47:51,409 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:47:51,409 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 14:47:51,670 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 14:47:51,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-gkhx.c.gcp-upstream.internal','N') ;"
2026-09-28 14:47:52,169 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:47:52,456 [I] Table x_portal_user does not exist in database ranger
2026-09-28 14:47:52,456 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:47:52,742 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 14:47:52,742 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:47:53,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:47:53,327 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 14:47:53,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 14:47:54.544 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.566 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.588 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.611 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.637 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.661 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.731 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.740 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.758 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.772 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.785 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.802 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.815 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.824 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.833 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:54.842 UTC [26211] WARNING: there is no transaction in progress
2026-09-28 14:47:55,746 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 14:47:55,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 14:47:55,999 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 14:47:55,999 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 14:47:56,304 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 14:47:56,305 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 14:47:56,555 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 14:47:56,556 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 14:47:56,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 14:47:57,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-gkhx.c.gcp-upstream.internal';"
2026-09-28 14:47:57,319 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 14:47:57,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 14:47:57,585 [I] DB_PATCHES have already been applied
I20260928 14:47:57.592118 14205 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-tasktMPbyY/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 14:47:57.592197 14205 mini_ranger.cc:193] Using host: 127.13.223.84
I20260928 14:47:57.596481 14205 mini_ranger.cc:227] Ranger admin URL: http://127.13.223.84:53585
Sep 28, 2026 2:47:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:47:57 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:47:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 2:47:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 2:47:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 2:47:58 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 2:47:58 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 14:47:58.735641 14205 mini_ranger.cc:160] Time spent starting Ranger: real 9.648s user 0.007s sys 0.003s
Sep 28, 2026 2:48:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 2:48:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 2:48:22 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 2:48:22 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 2:48:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 14:48:24.040295 14205 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.13.223.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.13.223.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangerkms_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.13.223.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/rangerkms_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.13.223.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.13.223.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.13.223.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/HTTP_127.13.223.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 14:48:24 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 14:48:24.156823 14205 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 14:48:24.171217 14205 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 14:48:25.726615 14205 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 14:48:25.726716 14205 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms
I20260928 14:48:25.726738 14205 mini_ranger_kms.cc:227] Using postgres at 127.13.223.84:38395
2026-09-28 14:48:25,970 [I] DB FLAVOR :POSTGRES
2026-09-28 14:48:25,971 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 14:48:25,971 [I] Checking connection
2026-09-28 14:48:25,971 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 14:48:26,284 [I] connection success
2026-09-28 14:48:26,285 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 14:48:26,285 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 14:48:26,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 14:48:26,605 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 14:48:26,606 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 14:48:26,606 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 14:48:26,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.13.223.84:38395/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 14:48:26,930 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 14:48:27.557071 14205 mini_ranger_kms.cc:310] Created kms service
I20260928 14:48:27.800751 14205 mini_ranger_kms.cc:326] Created kudu user
I20260928 14:48:27.866632 14205 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 14:48:27 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606904, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:28.365065 14205 mini_ranger_kms.cc:384] Added ranger policy
I20260928 14:48:28.365233 14205 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms:/tmp/dist-test-tasktMPbyY/build/debug/bin/postgresql.jar:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-tasktMPbyY/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 14:48:28.365257 14205 mini_ranger_kms.cc:240] Using host: 127.13.223.84
I20260928 14:48:28.369163 14205 mini_ranger_kms.cc:276] Ranger KMS PID: 26392
I20260928 14:48:28.369270 14205 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.13.223.84:46795
Sep 28, 2026 2:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 2:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 2:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 2:48:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 2:48:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 2:48:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 14:48:30.765053 14205 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.608s user 0.007s sys 0.220s
I20260928 14:48:30.765172 14205 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 14:48:30.765237 14205 mini_ranger_kms.cc:401] 127.13.223.84:46795/kms/v1/keys
Sep 28 14:48:31 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 14:48:31 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
Sep 28 14:48:31 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.13.223.84@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
Sep 28 14:48:34 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:48:34.796923 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42555
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:42555
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:34.938165 26442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:34.938521 26442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:34.938658 26442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:34.943087 26442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:34.943187 26442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:34.943212 26442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:34.943229 26442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:34.943246 26442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:34.949767 26442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
--ranger_kms_url=127.13.223.84:46795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:42555
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:34.951045 26442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:34.952167 26442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:34.964778 26447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:34.964775 26450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:34.964979 26448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:34.965034 26442 server_base.cc:1034] running on GCE node
I20260928 14:48:34.966120 26442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:34.967670 26442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:34.969017 26442 hybrid_clock.cc:648] HybridClock initialized: now 1790606914968895 us; error 141 us; skew 500 ppm
Sep 28 14:48:34 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:34.989842 26442 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:34.991525 26442 webserver.cc:460] Webserver started at http://127.13.223.126:43151/ using document root <none> and password file <none>
I20260928 14:48:34.992352 26442 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:34.992496 26442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:34.992780 26442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:48:34 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606914, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:35.084908 26442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "8c168ed36123410891c337f2bdea063f"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-gkhx"
server_key: "cd396b98a8470a96b941b925f172c384"
server_key_iv: "7d8b801a59d1ca4aee62bcb952bce694"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:35.085752 26442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "8c168ed36123410891c337f2bdea063f"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-gkhx"
server_key: "cd396b98a8470a96b941b925f172c384"
server_key_iv: "7d8b801a59d1ca4aee62bcb952bce694"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:35.092048 26442 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260928 14:48:35.111351 26457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:35.113292 26442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:35.113531 26442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "8c168ed36123410891c337f2bdea063f"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-gkhx"
server_key: "cd396b98a8470a96b941b925f172c384"
server_key_iv: "7d8b801a59d1ca4aee62bcb952bce694"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:35.113719 26442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:35.138816 26442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:35.142388 26442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:35.142701 26442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:35.152562 26442 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:42555
I20260928 14:48:35.152562 26509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:42555 every 8 connection(s)
I20260928 14:48:35.153919 26442 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:35.158025 26510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:35.160107 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 26442
I20260928 14:48:35.160375 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:48:35.160836 14205 external_mini_cluster.cc:1428] Setting key e71341b2826d20bc936b930fdb58e9ae
I20260928 14:48:35.167531 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Bootstrap starting.
I20260928 14:48:35.171304 26510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Neither blocks nor log segments found. Creating new log.
Sep 28 14:48:35 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:35.172421 26510 log.cc:824] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:35.176162 26510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: No bootstrap required, opened a new log
I20260928 14:48:35.180035 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:35.162738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.13.223.126:42555)
0928 14:48:35.163176 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:35.163198 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:35.163247 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:48:35.163941 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:35.163961 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:35.164350 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:35.164755 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:35.165770 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.167053 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.168156 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:35.168730 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:35.168958 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:35.172664 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:35.172671 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:35.172689 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:35.172842 (+ 153us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:35.176300 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.176967 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.176976 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.176982 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.177127 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:35.177569 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:35.177574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:35.177576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:35.178074 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:35.178323 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:35.178738 (+ 415us) server_negotiation.cc:299] Negotiation successful
0928 14:48:35.179110 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":133,"threads_started":1}
I20260928 14:48:35.180751 26510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:35.181139 26510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:35.181216 26510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c168ed36123410891c337f2bdea063f, State: Initialized, Role: FOLLOWER
I20260928 14:48:35.181823 26510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:35.182001 26510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:35.182068 26510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:35.182159 26510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:35.183573 26510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:35.184010 26510 leader_election.cc:302] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c168ed36123410891c337f2bdea063f; no voters:
I20260928 14:48:35.184366 26510 leader_election.cc:288] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:35.184554 26515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:35.184790 26515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 LEADER]: Becoming Leader. State: Replica: 8c168ed36123410891c337f2bdea063f, State: Running, Role: LEADER
I20260928 14:48:35.185125 26515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:35.185738 26510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:35.186529 26515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c168ed36123410891c337f2bdea063f. Latest consensus state: current_term: 1 leader_uuid: "8c168ed36123410891c337f2bdea063f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } } }
I20260928 14:48:35.186679 26515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:35.187269 26523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:35.187526 26516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c168ed36123410891c337f2bdea063f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } } }
I20260928 14:48:35.187618 26516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:35.191123 26523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:35.191625 26510 ranger_client.cc:318] Using new properties file: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:48:35.201885 26523 catalog_manager.cc:1347] Generated new cluster ID: bd41408483e64d29a1a14f668f142e05
I20260928 14:48:35.202000 26523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:35.243892 26523 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:35.245579 26523 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:35.293857 26523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Generated new TSK 0
I20260928 14:48:35.295090 26523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:48:36.072432 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:36.237284 26555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:36.237633 26555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:36.237716 26555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:36.242352 26555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:36.242465 26555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:36.242612 26555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:48:36.248713 26555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26555
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:36.250042 26555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:36.251288 26555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:36.262717 26562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:36.262811 26563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:36.262873 26565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:36.264289 26555 server_base.cc:1034] running on GCE node
I20260928 14:48:36.264834 26555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:36.265578 26555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:36.268937 26555 hybrid_clock.cc:648] HybridClock initialized: now 1790606916268827 us; error 105 us; skew 500 ppm
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:36.275802 26555 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:48:36.278834 26555 webserver.cc:460] Webserver started at http://127.13.223.65:34793/ using document root <none> and password file <none>
I20260928 14:48:36.279709 26555 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:36.279838 26555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:36.280149 26555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:36.299784 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "ca3eeddd8d324d40b363708480152500"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "4d518bc56a8ee18baa8b10dfab76a577"
server_key_iv: "460e7c2583600d9592b10dcd65a14e1c"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.301052 26555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "ca3eeddd8d324d40b363708480152500"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "4d518bc56a8ee18baa8b10dfab76a577"
server_key_iv: "460e7c2583600d9592b10dcd65a14e1c"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.322068 26555 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.009s sys 0.000s
I20260928 14:48:36.344959 26573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.347543 26555 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:48:36.347754 26555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "ca3eeddd8d324d40b363708480152500"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "4d518bc56a8ee18baa8b10dfab76a577"
server_key_iv: "460e7c2583600d9592b10dcd65a14e1c"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.347937 26555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:36.382197 26555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:36.395635 26555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:36.395866 26555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:36.396631 26555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:36.398173 26555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:36.398249 26555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.398311 26555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:36.398339 26555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.411548 26555 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:35657
I20260928 14:48:36.413080 26555 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:36.414126 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 26555
I20260928 14:48:36.414279 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:36.414649 14205 external_mini_cluster.cc:1428] Setting key 677ba1ef40a4cba180a13af5815c8f5d
I20260928 14:48:36.424655 26687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:35657 every 8 connection(s)
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:48:36.462188 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:36.430824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:42291 (local address 127.13.223.126:42555)
0928 14:48:36.434407 (+ 3583us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:36.434413 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:36.434431 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:48:36.434484 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:36.434487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:36.434585 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:36.434688 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:36.436626 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.437618 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.440037 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.440487 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.440612 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:36.455022 (+ 14410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:36.455032 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:36.455038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:36.455087 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:36.458445 (+ 3358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:36.459452 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:36.459459 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:36.459461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:36.459594 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:36.460207 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:36.460214 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:36.460218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:36.460565 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:36.460682 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:36.461717 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0928 14:48:36.461911 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3471,"thread_start_us":82,"threads_started":1}
I20260928 14:48:36.465431 26691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:36.429978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.65:42291)
0928 14:48:36.432428 (+ 2450us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:36.432511 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:36.433802 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:36.435084 (+ 1282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:36.435096 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:36.435555 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:36.436388 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:36.436413 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.437814 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.437819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:36.439816 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:36.439834 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.444506 (+ 4672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.444513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:36.444901 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:36.445792 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:36.445818 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:36.454654 (+ 8836us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:36.458740 (+ 4086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:36.458748 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:36.458771 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:36.459256 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:36.459758 (+ 502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:36.459765 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:36.459771 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:36.460045 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:36.460818 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:36.460829 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:36.461304 (+ 475us) client_negotiation.cc:769] Sending connection context
0928 14:48:36.462434 (+ 1130us) client_negotiation.cc:240] Negotiation successful
0928 14:48:36.462983 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2138,"thread_start_us":239,"threads_started":1}
I20260928 14:48:36.467765 26688 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:36.468142 26688 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:36.468930 26688 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:36.471292 26474 ts_manager.cc:194] Registered new tserver with Master: ca3eeddd8d324d40b363708480152500 (127.13.223.65:35657)
I20260928 14:48:36.473292 26474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:42291
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:48:36.486477 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:36.696821 26694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:36.697350 26694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:36.697477 26694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:36.704849 26694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:36.705097 26694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:36.705264 26694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:48:36.714782 26694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:36.716627 26694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:36.718361 26694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:36.734382 26701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:36.734469 26700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:36.734382 26703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:36.735447 26694 server_base.cc:1034] running on GCE node
I20260928 14:48:36.736202 26694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:36.737190 26694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:36.740568 26694 hybrid_clock.cc:648] HybridClock initialized: now 1790606916740472 us; error 117 us; skew 500 ppm
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:36.745009 26694 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:48:36.746445 26694 webserver.cc:460] Webserver started at http://127.13.223.66:35117/ using document root <none> and password file <none>
I20260928 14:48:36.747387 26694 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:36.747565 26694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:36.747880 26694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:36.772710 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "1053bd4701720eaa509a161f5eb07c27"
server_key_iv: "4861c3aeb7fb43ea42abebfa66518af5"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.773999 26694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "1053bd4701720eaa509a161f5eb07c27"
server_key_iv: "4861c3aeb7fb43ea42abebfa66518af5"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.787595 26694 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260928 14:48:36.812197 26710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.814013 26694 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:48:36.814188 26694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "1053bd4701720eaa509a161f5eb07c27"
server_key_iv: "4861c3aeb7fb43ea42abebfa66518af5"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:36.814347 26694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:36.830999 26694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:36.835011 26694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:36.835237 26694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:36.836014 26694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:36.837144 26694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:36.837224 26694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.837277 26694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:36.837299 26694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:36.849615 26694 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:45577
I20260928 14:48:36.849833 26823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:45577 every 8 connection(s)
I20260928 14:48:36.851151 26694 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:36.857405 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 26694
I20260928 14:48:36.857576 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:36.857981 14205 external_mini_cluster.cc:1428] Setting key 3a79976d2b5824807ab03c35749a560d
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Sep 28 14:48:36 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:48:36.906431 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:36.855827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:39687 (local address 127.13.223.126:42555)
0928 14:48:36.855983 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:36.855989 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:36.856010 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:48:36.859061 (+ 3051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:36.859067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:36.859145 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:36.859247 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:36.861000 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.861791 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.878940 (+ 17149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.879371 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.879528 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:36.900153 (+ 20625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:36.900160 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:36.900164 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:36.900220 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:36.902599 (+ 2379us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:36.903880 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:36.903886 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:36.903888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:36.904025 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:36.904625 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:36.904630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:36.904632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:36.904930 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:36.905043 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:36.906000 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 14:48:36.906203 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":49}
I20260928 14:48:36.908699 26826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:36.854480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.66:39687)
0928 14:48:36.855384 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:36.855475 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:36.856935 (+ 1460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:36.859527 (+ 2592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:36.859539 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:36.859975 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:36.860776 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:36.860821 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.861985 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.861988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:36.878514 (+ 16526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:36.878543 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:36.879811 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:36.879818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:36.880106 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:36.892991 (+ 12885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:36.893017 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:36.899814 (+ 6797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:36.902849 (+ 3035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:36.902859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:36.902884 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:36.903684 (+ 800us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:36.904192 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:36.904195 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:36.904198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:36.904478 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:36.905137 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:36.905147 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:36.905622 (+ 475us) client_negotiation.cc:769] Sending connection context
0928 14:48:36.906838 (+ 1216us) client_negotiation.cc:240] Negotiation successful
0928 14:48:36.907345 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":223,"threads_started":1}
I20260928 14:48:36.910631 26824 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:36.910995 26824 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:36.911826 26824 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:36.913362 26474 ts_manager.cc:194] Registered new tserver with Master: c6b0a29ecd8541bd8951b5fa2ac02a48 (127.13.223.66:45577)
I20260928 14:48:36.914289 26474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:39687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:48:36.943840 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606916, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
W20260928 14:48:37.150238 26829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:37.151039 26829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:37.151119 26829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:37.157835 26829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:37.157992 26829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:37.158111 26829 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:48:37.167273 26829 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26829
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:37.168948 26829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:37.170406 26829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:37.185108 26835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:37.185052 26837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:37.185148 26834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:37.185652 26829 server_base.cc:1034] running on GCE node
I20260928 14:48:37.186398 26829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:37.187340 26829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:37.188721 26829 hybrid_clock.cc:648] HybridClock initialized: now 1790606917188652 us; error 94 us; skew 500 ppm
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:37.193035 26829 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:48:37.194387 26829 webserver.cc:460] Webserver started at http://127.13.223.67:45681/ using document root <none> and password file <none>
I20260928 14:48:37.195209 26829 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:37.195369 26829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:37.195660 26829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:37.214102 26829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "d09a957693ac4349b50949d5db0992bb"
format_stamp: "Formatted at 2026-09-28 14:48:37 on dist-test-slave-gkhx"
server_key: "cdc0d8d9cb7be86e81f2b7046e719e5f"
server_key_iv: "4f5b86de613163beca6109edc2851e52"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:37.215188 26829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "d09a957693ac4349b50949d5db0992bb"
format_stamp: "Formatted at 2026-09-28 14:48:37 on dist-test-slave-gkhx"
server_key: "cdc0d8d9cb7be86e81f2b7046e719e5f"
server_key_iv: "4f5b86de613163beca6109edc2851e52"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:37.227384 26829 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.005s sys 0.000s
I20260928 14:48:37.251013 26844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.252991 26829 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260928 14:48:37.253145 26829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "d09a957693ac4349b50949d5db0992bb"
format_stamp: "Formatted at 2026-09-28 14:48:37 on dist-test-slave-gkhx"
server_key: "cdc0d8d9cb7be86e81f2b7046e719e5f"
server_key_iv: "4f5b86de613163beca6109edc2851e52"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:37.253297 26829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:37.275094 26829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:37.279247 26829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:37.279461 26829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:37.280279 26829 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:37.281442 26829 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:37.281512 26829 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.281571 26829 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:37.281594 26829 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.297854 26829 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:45337
I20260928 14:48:37.298180 26957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:45337 every 8 connection(s)
I20260928 14:48:37.299356 26829 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:37.305015 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 26829
I20260928 14:48:37.305160 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:37.305506 14205 external_mini_cluster.cc:1428] Setting key e7eaf2f3e151c244abd89d2e445bb475
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:37.324937 26692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:37.302313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:39763 (local address 127.13.223.126:42555)
0928 14:48:37.302478 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:37.302483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:37.303884 (+ 1401us) server_negotiation.cc:407] Connection header received
0928 14:48:37.305790 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:37.305794 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:37.305857 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:37.305909 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:37.309793 (+ 3884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.310727 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.312852 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.313262 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.313401 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:37.317502 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:37.317507 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:37.317510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:37.317545 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:37.321011 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:37.322136 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:37.322141 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:37.322143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:37.322427 (+ 284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:37.323515 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:37.323520 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:37.323521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:37.323811 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:37.323907 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:37.324583 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 14:48:37.324754 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260928 14:48:37.325966 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:37.302882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.67:39763)
0928 14:48:37.303694 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:37.303760 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:37.305440 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:37.307011 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:37.307022 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:37.308324 (+ 1302us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:37.309370 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:37.309585 (+ 215us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.310902 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.310906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:37.312646 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:37.312662 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.313482 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.313486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:37.313805 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:37.314780 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:37.314805 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:37.317221 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:37.321286 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:37.321295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:37.321319 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:37.321909 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:37.323011 (+ 1102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:37.323016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:37.323018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:37.323350 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:37.323985 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:37.323991 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:37.324302 (+ 311us) client_negotiation.cc:769] Sending connection context
0928 14:48:37.324522 (+ 220us) client_negotiation.cc:240] Negotiation successful
0928 14:48:37.324903 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":283,"threads_started":1}
I20260928 14:48:37.327634 26958 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:37.328060 26958 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:37.328783 26958 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:37.330159 26474 ts_manager.cc:194] Registered new tserver with Master: d09a957693ac4349b50949d5db0992bb (127.13.223.67:45337)
I20260928 14:48:37.331099 26474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:39763
I20260928 14:48:37.333073 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:37.333844 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 26555
I20260928 14:48:37.343626 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 26694
I20260928 14:48:37.350596 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 26829
I20260928 14:48:37.356252 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 26442
I20260928 14:48:37.365018 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42555
--webserver_interface=127.13.223.126
--webserver_port=43151
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:42555
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:37.488628 26961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:37.488942 26961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:37.488987 26961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:37.493367 26961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:37.493474 26961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:37.493495 26961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:37.493512 26961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:37.493531 26961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:37.499599 26961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:42555
--ranger_config_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=43151
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:37.500840 26961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:37.501966 26961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:37.511615 26967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:37.511669 26966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:37.511998 26961 server_base.cc:1034] running on GCE node
W20260928 14:48:37.511617 26969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:37.513051 26961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:37.514268 26961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:37.515491 26961 hybrid_clock.cc:648] HybridClock initialized: now 1790606917515469 us; error 44 us; skew 500 ppm
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:37.518890 26961 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:37.520403 26961 webserver.cc:460] Webserver started at http://127.13.223.126:43151/ using document root <none> and password file <none>
I20260928 14:48:37.521242 26961 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:37.521375 26961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:37.590806 26961 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:48:37.593556 26977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.595355 26961 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:37.595507 26961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "8c168ed36123410891c337f2bdea063f"
format_stamp: "Formatted at 2026-09-28 14:48:35 on dist-test-slave-gkhx"
server_key: "cd396b98a8470a96b941b925f172c384"
server_key_iv: "7d8b801a59d1ca4aee62bcb952bce694"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:37.596145 26961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:37.624323 26961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:37.628038 26961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:37.628312 26961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:37.637084 26961 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:42555
I20260928 14:48:37.637091 27029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:42555 every 8 connection(s)
I20260928 14:48:37.638399 26961 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:37.643306 27030 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:37.645220 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 26961
I20260928 14:48:37.645390 27030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Bootstrap starting.
I20260928 14:48:37.645776 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:35657
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:42555
--webserver_port=34793
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:37.664162 27030 log.cc:824] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:37.671509 27030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:37.671991 27030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Bootstrap complete.
I20260928 14:48:37.676496 27030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:37.677081 27030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c168ed36123410891c337f2bdea063f, State: Initialized, Role: FOLLOWER
I20260928 14:48:37.677703 27030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:37.677835 27030 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:37.677924 27030 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:37.678027 27030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:48:37.680351 27030 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:37.680848 27030 leader_election.cc:302] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c168ed36123410891c337f2bdea063f; no voters:
I20260928 14:48:37.681226 27030 leader_election.cc:288] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:48:37.681543 27034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:48:37.681927 27034 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [term 2 LEADER]: Becoming Leader. State: Replica: 8c168ed36123410891c337f2bdea063f, State: Running, Role: LEADER
I20260928 14:48:37.682324 27034 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } }
I20260928 14:48:37.682732 27030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:37.684139 27036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c168ed36123410891c337f2bdea063f. Latest consensus state: current_term: 2 leader_uuid: "8c168ed36123410891c337f2bdea063f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } } }
I20260928 14:48:37.684185 27035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "8c168ed36123410891c337f2bdea063f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c168ed36123410891c337f2bdea063f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42555 } } }
I20260928 14:48:37.684286 27036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:37.684302 27035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:37.684631 27043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:37.688624 27043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:37.689332 27030 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 14:48:37.689395 27043 catalog_manager.cc:1259] Loaded cluster ID: bd41408483e64d29a1a14f668f142e05
I20260928 14:48:37.689451 27043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:37.695026 27043 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:37.696275 27043 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Loaded TSK: 0
I20260928 14:48:37.697856 27043 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:48:37.788877 27032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:37.789328 27032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:37.789409 27032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:37.796852 27032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:37.796969 27032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:37.797076 27032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:48:37.806284 27032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:35657
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=34793
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:37.807871 27032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:37.809365 27032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:37.824087 27070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:37.824512 27067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:37.824693 27068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:37.824956 27032 server_base.cc:1034] running on GCE node
I20260928 14:48:37.825554 27032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:37.826458 27032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:37.827636 27032 hybrid_clock.cc:648] HybridClock initialized: now 1790606917827613 us; error 43 us; skew 500 ppm
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:37.831751 27032 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:48:37.833083 27032 webserver.cc:460] Webserver started at http://127.13.223.65:34793/ using document root <none> and password file <none>
I20260928 14:48:37.833877 27032 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:37.833992 27032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:37.870201 27032 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:37.873675 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.875305 27032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:48:37.875479 27032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "ca3eeddd8d324d40b363708480152500"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "4d518bc56a8ee18baa8b10dfab76a577"
server_key_iv: "460e7c2583600d9592b10dcd65a14e1c"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:37.876255 27032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:37.893851 27032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:37.898746 27032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:37.899075 27032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:37.899977 27032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:37.901273 27032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:37.901394 27032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.901541 27032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:37.901623 27032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:37.913995 27032 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:35657
I20260928 14:48:37.914275 27191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:35657 every 8 connection(s)
I20260928 14:48:37.915699 27032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:37.917162 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27032
I20260928 14:48:37.917819 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:45577
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:42555
--webserver_port=35117
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:37 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606917, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:37.941313 27195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:37.918829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.65:46325)
0928 14:48:37.920425 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:37.920500 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:37.921759 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:37.924115 (+ 2356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:37.924127 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:37.924547 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:37.925324 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:37.925351 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.927601 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.927607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:37.929386 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:37.929405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.930836 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.930841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:37.931076 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:37.931899 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:37.931922 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:37.933565 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:37.936636 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:37.936644 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:37.936664 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:37.937104 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:37.937650 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:37.937654 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:37.937656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:37.937883 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:37.938911 (+ 1028us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:37.938918 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:37.939306 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 14:48:37.939617 (+ 311us) client_negotiation.cc:240] Negotiation successful
0928 14:48:37.940049 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1315,"thread_start_us":163,"threads_started":1}
I20260928 14:48:37.943722 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:37.918607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:46325 (local address 127.13.223.126:42555)
0928 14:48:37.922450 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:37.922478 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:37.922533 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:48:37.922959 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:37.922979 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:37.923548 (+ 569us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:37.923972 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:37.925588 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.927361 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.929603 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:37.930536 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:37.930797 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:37.933913 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:37.933922 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:37.933943 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:37.933985 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:37.936440 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:37.937311 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:37.937321 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:37.937328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:37.937514 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:37.938014 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:37.938018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:37.938020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:37.938692 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:37.938834 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:37.942194 (+ 3360us) server_negotiation.cc:299] Negotiation successful
0928 14:48:37.942624 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3611,"thread_start_us":143,"threads_started":1}
I20260928 14:48:37.947113 27192 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:37.947469 27192 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:37.948298 27192 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:37.951195 27192 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:42555 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:48:37.951371 27192 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:37.951622 27192 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:37.952728 27192 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:37.952963 27192 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:37.953882 27192 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:48:38.102646 27196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:38.103106 27196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:38.103199 27196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:38.108536 27196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:38.108614 27196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:38.108681 27196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:48:38.116088 27196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:45577
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=35117
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:38.117517 27196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:38.118774 27196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:38.132678 27202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:38.132678 27201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:38.133009 27196 server_base.cc:1034] running on GCE node
W20260928 14:48:38.133157 27204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:38.133782 27196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:38.134641 27196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:38.135946 27196 hybrid_clock.cc:648] HybridClock initialized: now 1790606918135909 us; error 58 us; skew 500 ppm
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:38.140300 27196 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:48:38.141660 27196 webserver.cc:460] Webserver started at http://127.13.223.66:35117/ using document root <none> and password file <none>
I20260928 14:48:38.142457 27196 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:38.142608 27196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:38.165256 27196 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:38.169111 27212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.171078 27196 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:48:38.171325 27196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48"
format_stamp: "Formatted at 2026-09-28 14:48:36 on dist-test-slave-gkhx"
server_key: "1053bd4701720eaa509a161f5eb07c27"
server_key_iv: "4861c3aeb7fb43ea42abebfa66518af5"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:38.172271 27196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:38.193540 27196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:38.197791 27196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:38.198029 27196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:38.198923 27196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:38.200160 27196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:38.200237 27196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.200304 27196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:38.200328 27196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.213387 27196 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:45577
I20260928 14:48:38.213465 27325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:45577 every 8 connection(s)
I20260928 14:48:38.214998 27196 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:38.224851 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27196
I20260928 14:48:38.225350 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.13.223.84:46795/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:45337
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:42555
--webserver_port=45681
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:38.235401 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:38.219478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:39461 (local address 127.13.223.126:42555)
0928 14:48:38.219623 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:38.219628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:38.219649 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:48:38.220365 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:38.220370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:38.220446 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:38.220563 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:38.222811 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.223672 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.225999 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.226412 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.226602 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:38.230418 (+ 3816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:38.230422 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:38.230424 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:38.230456 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:38.232301 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.233028 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.233032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.233034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.233136 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.233565 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.233569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.233571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.233812 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:38.233934 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:38.235002 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0928 14:48:38.235192 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:48:38.236284 27328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:38.218112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.66:39461)
0928 14:48:38.218824 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:38.218892 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:38.220045 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:38.220945 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:38.220956 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:38.221381 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:38.222535 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:38.222604 (+ 69us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.224087 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.224091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:38.225780 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:38.225801 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.226871 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.226877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:38.227162 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:38.228017 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:38.228042 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:38.230131 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:38.232522 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:38.232528 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:38.232547 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:38.232886 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:38.233248 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:38.233253 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:38.233255 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:38.233444 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:38.233937 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:38.233948 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:38.234636 (+ 688us) client_negotiation.cc:769] Sending connection context
0928 14:48:38.234914 (+ 278us) client_negotiation.cc:240] Negotiation successful
0928 14:48:38.235334 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":202,"threads_started":1}
I20260928 14:48:38.237602 27326 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:38.237913 27326 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.238499 27326 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:38.240839 27326 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:42555 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:48:38.241009 27326 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.241269 27326 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:38.242300 27326 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.242533 27326 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:38.243449 27326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:48:38.399869 27329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:38.400286 27329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:38.400388 27329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:38.406867 27329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:38.406967 27329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:38.407070 27329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:48:38.416275 27329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.13.223.84:46795/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:45337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=45681
--tserver_master_addrs=127.13.223.126:42555
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:38.417732 27329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:38.419067 27329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:38.430820 27338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:38.430822 27336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:38.432025 27335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:38.432896 27329 server_base.cc:1034] running on GCE node
I20260928 14:48:38.433627 27329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:38.434711 27329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:38.436105 27329 hybrid_clock.cc:648] HybridClock initialized: now 1790606918436058 us; error 59 us; skew 500 ppm
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:38.439610 27329 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:48:38.440900 27329 webserver.cc:460] Webserver started at http://127.13.223.67:45681/ using document root <none> and password file <none>
I20260928 14:48:38.441558 27329 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:38.441694 27329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:38.464525 27329 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:48:38.467830 27346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.469596 27329 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:38.469785 27329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "d09a957693ac4349b50949d5db0992bb"
format_stamp: "Formatted at 2026-09-28 14:48:37 on dist-test-slave-gkhx"
server_key: "cdc0d8d9cb7be86e81f2b7046e719e5f"
server_key_iv: "4f5b86de613163beca6109edc2851e52"
server_key_version: "kuduclusterkey@0"
I20260928 14:48:38.470482 27329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:38.512312 27329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:38.516402 27329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:38.516724 27329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:38.517547 27329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:38.518898 27329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:38.519016 27329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.519129 27329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:38.519202 27329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:38.532130 27329 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:45337
I20260928 14:48:38.532218 27459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:45337 every 8 connection(s)
I20260928 14:48:38.533785 27329 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:38.539245 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27329
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:38.554734 27464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:38.540344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.13.223.126:42555)
0928 14:48:38.540617 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:38.540622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:38.542374 (+ 1752us) server_negotiation.cc:407] Connection header received
0928 14:48:38.542539 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:38.542543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:38.542639 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:38.542748 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:38.543578 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.544261 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.545596 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.545936 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.546049 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:38.548048 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:38.548055 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:38.548061 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:38.548107 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:38.550895 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.551784 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.551789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.551791 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.551886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.553173 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.553177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.553178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.553595 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:38.553826 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:38.554104 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:48:38.554455 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":173,"spinlock_wait_cycles":1792,"thread_start_us":90,"threads_started":1}
I20260928 14:48:38.555827 27194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:38.536654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44349 (local address 127.13.223.126:42555)
0928 14:48:38.536844 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:38.536850 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:38.540969 (+ 4119us) server_negotiation.cc:407] Connection header received
0928 14:48:38.541994 (+ 1025us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:38.541999 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:38.542062 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:38.542166 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:38.544039 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.544641 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.546190 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.546409 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.546513 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:38.549896 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:38.549901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:38.549904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:38.549965 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:38.551784 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.552570 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.552574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.552575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.552664 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:38.553163 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:38.553167 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:38.553169 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:38.553592 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:38.553715 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:38.555485 (+ 1770us) server_negotiation.cc:299] Negotiation successful
0928 14:48:38.555641 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":32384}
I20260928 14:48:38.557850 27462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:38.537230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42555 (local address 127.13.223.67:44349)
0928 14:48:38.540710 (+ 3480us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:38.540815 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:38.541743 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:38.542575 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:38.542589 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:38.543020 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:38.543853 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:38.543881 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.544788 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.544792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:38.546057 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:38.546069 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:38.546775 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:38.546782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:38.546980 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:38.547607 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:38.547626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:38.549473 (+ 1847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:38.551945 (+ 2472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:38.551954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:38.551977 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:38.552416 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:38.552824 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:38.552828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:38.552830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:38.553034 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:38.554455 (+ 1421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:38.554464 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:38.555108 (+ 644us) client_negotiation.cc:769] Sending connection context
0928 14:48:38.556064 (+ 956us) client_negotiation.cc:240] Negotiation successful
0928 14:48:38.556497 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3200,"thread_start_us":183,"threads_started":1}
I20260928 14:48:38.559759 27460 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42555
I20260928 14:48:38.560081 27460 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.560787 27460 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:38.562891 27460 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:42555 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:48:38.563076 27460 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.563305 27460 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:38.565474 27460 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.565753 27460 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
W20260928 14:48:38.566877 27460 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:48:38.954210 27192 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:38.954891 27192 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:38.956281 26993 ts_manager.cc:194] Registered new tserver with Master: ca3eeddd8d324d40b363708480152500 (127.13.223.65:35657)
I20260928 14:48:38.957914 26993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:46325
Sep 28 14:48:38 dist-test-slave-gkhx krb5kdc[25998](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:39.243760 27326 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:39.244138 27326 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:39.245105 26993 ts_manager.cc:194] Registered new tserver with Master: c6b0a29ecd8541bd8951b5fa2ac02a48 (127.13.223.66:45577)
I20260928 14:48:39.245935 26993 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:39461
Sep 28 14:48:39 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
I20260928 14:48:39.567163 27460 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:39.567529 27460 heartbeater.cc:511] Master 127.13.223.126:42555 requested a full tablet report, sending...
I20260928 14:48:39.568655 26994 ts_manager.cc:194] Registered new tserver with Master: d09a957693ac4349b50949d5db0992bb (127.13.223.67:45337)
I20260928 14:48:39.569563 26994 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:44349
I20260928 14:48:39.574465 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:39.592322 27482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.576606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.13.223.126:42555)
0928 14:48:39.576942 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:39.576948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:39.579054 (+ 2106us) server_negotiation.cc:407] Connection header received
0928 14:48:39.579228 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:39.579232 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:39.579295 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:39.579402 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:39.580546 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.581777 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.583207 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.583640 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.583799 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.586036 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:39.586041 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:39.586044 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:39.586085 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:39.588769 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:39.589684 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:39.589689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:39.589691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:39.589847 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:39.590640 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:39.590646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:39.590649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:39.591077 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:39.591235 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:39.591791 (+ 556us) server_negotiation.cc:299] Negotiation successful
0928 14:48:39.592013 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":243,"thread_start_us":134,"threads_started":1}
I20260928 14:48:39.599967 26994 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:39.606700 26994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:39.632027 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.619037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38798 (local address 127.13.223.65:35657)
0928 14:48:39.622245 (+ 3208us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:39.622251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:39.622275 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:48:39.622383 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:39.622389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:39.622708 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:39.622882 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:39.623454 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.624313 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.627596 (+ 3283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.628800 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.628900 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.629082 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:39.629803 (+ 721us) server_negotiation.cc:299] Negotiation successful
0928 14:48:39.629946 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3097,"thread_start_us":130,"threads_started":1}
I20260928 14:48:39.632516 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.618928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:35657 (local address 127.0.0.1:38798)
0928 14:48:39.621686 (+ 2758us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:39.621758 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:39.621917 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:39.623091 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:39.623099 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:39.623129 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:39.623292 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.623302 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.624456 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.624463 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.627399 (+ 2936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.627413 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.629193 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.629197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.629519 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.629575 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 14:48:39.632271 (+ 2696us) client_negotiation.cc:240] Negotiation successful
0928 14:48:39.632354 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2520,"thread_start_us":141,"threads_started":1}
I20260928 14:48:39.632536 27489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.620407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:45577 (local address 127.0.0.1:54218)
0928 14:48:39.624639 (+ 4232us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:39.624656 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:39.624900 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:39.626138 (+ 1238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:39.626142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:39.626166 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:39.626284 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.626292 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.628076 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.628080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.629945 (+ 1865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.629956 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.631552 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.631557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.631816 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.631847 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:48:39.631919 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:48:39.632112 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4150,"thread_start_us":115,"threads_started":1}
I20260928 14:48:39.635115 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.620572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.13.223.66:45577)
0928 14:48:39.625529 (+ 4957us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:39.625536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:39.625564 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:48:39.625652 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:39.625658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:39.625917 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:39.626468 (+ 551us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:39.626667 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.627931 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.630130 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.631382 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.634512 (+ 3130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.634762 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:39.634827 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 14:48:39.634948 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4823,"thread_start_us":129,"threads_started":1}
I20260928 14:48:39.637410 27126 tablet_service.cc:1467] Processing CreateTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d (DEFAULT_TABLE table=test-table [id=917f284511704b8ea89734a6d3fc6733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:39.638852 27487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.621815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45337 (local address 127.0.0.1:46330)
0928 14:48:39.632738 (+ 10923us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:39.632755 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:39.632957 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:39.633820 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:39.633823 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:39.633847 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:39.633939 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.633946 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.635584 (+ 1638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.635587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.637152 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:39.637165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.638324 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.638329 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:39.638434 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.638464 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:48:39.638583 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:48:39.638706 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10845,"thread_start_us":119,"threads_started":1}
I20260928 14:48:39.638962 27126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55367ddc094a41e4b46d9367cf9fb74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:39.638615 27260 tablet_service.cc:1467] Processing CreateTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d (DEFAULT_TABLE table=test-table [id=917f284511704b8ea89734a6d3fc6733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:39.639820 27260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55367ddc094a41e4b46d9367cf9fb74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:39.643702 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:39.621524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.13.223.67:45337)
0928 14:48:39.625293 (+ 3769us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:39.625301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:39.632856 (+ 7555us) server_negotiation.cc:407] Connection header received
0928 14:48:39.633138 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:39.633146 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:39.633433 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:39.633591 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:39.634093 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.635420 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.637319 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:39.638174 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:39.638300 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:39.638453 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:39.643335 (+ 4882us) server_negotiation.cc:299] Negotiation successful
0928 14:48:39.643524 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3557,"thread_start_us":102,"threads_started":1}
I20260928 14:48:39.647521 27394 tablet_service.cc:1467] Processing CreateTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d (DEFAULT_TABLE table=test-table [id=917f284511704b8ea89734a6d3fc6733]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:39.648984 27394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55367ddc094a41e4b46d9367cf9fb74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:39.650206 27493 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Bootstrap starting.
I20260928 14:48:39.652746 27493 tablet_bootstrap.cc:654] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:39.653625 27493 log.cc:824] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:39.654479 27494 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Bootstrap starting.
I20260928 14:48:39.656710 27494 tablet_bootstrap.cc:654] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:39.657532 27494 log.cc:824] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:39.660737 27497 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Bootstrap starting.
I20260928 14:48:39.660842 27494 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: No bootstrap required, opened a new log
I20260928 14:48:39.661072 27494 ts_tablet_manager.cc:1397] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260928 14:48:39.663858 27497 tablet_bootstrap.cc:654] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:39.665099 27497 log.cc:824] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:39.666178 27494 raft_consensus.cc:348] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.666540 27494 raft_consensus.cc:374] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:39.666651 27494 raft_consensus.cc:729] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6b0a29ecd8541bd8951b5fa2ac02a48, State: Initialized, Role: FOLLOWER
I20260928 14:48:39.668061 27493 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: No bootstrap required, opened a new log
I20260928 14:48:39.668061 27497 tablet_bootstrap.cc:492] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: No bootstrap required, opened a new log
I20260928 14:48:39.668300 27493 ts_tablet_manager.cc:1397] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.000s
I20260928 14:48:39.668300 27497 ts_tablet_manager.cc:1397] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 14:48:39.673430 27493 raft_consensus.cc:348] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.673834 27493 raft_consensus.cc:374] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:39.673899 27493 raft_consensus.cc:729] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca3eeddd8d324d40b363708480152500, State: Initialized, Role: FOLLOWER
I20260928 14:48:39.674522 27493 consensus_queue.cc:260] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.675485 27497 raft_consensus.cc:348] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.675859 27497 raft_consensus.cc:374] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:39.675958 27497 raft_consensus.cc:729] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d09a957693ac4349b50949d5db0992bb, State: Initialized, Role: FOLLOWER
I20260928 14:48:39.676033 27494 consensus_queue.cc:260] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.676676 27497 consensus_queue.cc:260] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:39.677570 27493 ts_tablet_manager.cc:1428] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Time spent starting tablet: real 0.009s user 0.002s sys 0.007s
I20260928 14:48:39.677855 27497 ts_tablet_manager.cc:1428] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260928 14:48:39.678135 27494 ts_tablet_manager.cc:1428] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Time spent starting tablet: real 0.017s user 0.009s sys 0.001s
I20260928 14:48:39.678326 27460 heartbeater.cc:503] Master 127.13.223.126:42555 was elected leader, sending a full tablet report...
I20260928 14:48:39.678474 27192 heartbeater.cc:503] Master 127.13.223.126:42555 was elected leader, sending a full tablet report...
I20260928 14:48:39.683247 27326 heartbeater.cc:503] Master 127.13.223.126:42555 was elected leader, sending a full tablet report...
W20260928 14:48:39.717733 27327 tablet.cc:2369] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:48:39.787746 27461 tablet.cc:2369] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 14:48:39 dist-test-slave-gkhx krb5kdc[25998](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606918, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for HTTP/127.13.223.84@KRBTEST.COM
W20260928 14:48:39.925014 27193 tablet.cc:2369] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:40.005262 27499 raft_consensus.cc:482] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:40.005587 27499 raft_consensus.cc:504] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:40.008100 27499 leader_election.cc:288] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6b0a29ecd8541bd8951b5fa2ac02a48 (127.13.223.66:45577), d09a957693ac4349b50949d5db0992bb (127.13.223.67:45337)
I20260928 14:48:40.026407 27490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.008805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:47623 (local address 127.13.223.66:45577)
0928 14:48:40.008913 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:40.008918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:40.017588 (+ 8670us) server_negotiation.cc:407] Connection header received
0928 14:48:40.017849 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:40.017853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:40.017906 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:40.017956 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:40.018644 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.019491 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.024795 (+ 5304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.025513 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.025649 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:40.025715 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:40.026115 (+ 400us) server_negotiation.cc:299] Negotiation successful
0928 14:48:40.026240 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:48:40.027334 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.008331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:45577 (local address 127.13.223.65:47623)
0928 14:48:40.017415 (+ 9084us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:40.017440 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:40.017703 (+ 263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:40.018106 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:40.018109 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:40.018127 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:40.018274 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:40.018280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.019684 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.019688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:40.021050 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:40.021066 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.025845 (+ 4779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.025849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:40.025963 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:40.025987 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 14:48:40.027024 (+ 1037us) client_negotiation.cc:240] Negotiation successful
0928 14:48:40.027147 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8957,"thread_start_us":132,"threads_started":1}
I20260928 14:48:40.028303 27271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55367ddc094a41e4b46d9367cf9fb74d" candidate_uuid: "ca3eeddd8d324d40b363708480152500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" is_pre_election: true
I20260928 14:48:40.028720 27271 raft_consensus.cc:2454] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3eeddd8d324d40b363708480152500 in term 0.
I20260928 14:48:40.029513 27082 leader_election.cc:302] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b0a29ecd8541bd8951b5fa2ac02a48, ca3eeddd8d324d40b363708480152500; no voters:
I20260928 14:48:40.029655 27491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.008610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:41669 (local address 127.13.223.67:45337)
0928 14:48:40.008753 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:40.008758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:40.021300 (+ 12542us) server_negotiation.cc:407] Connection header received
0928 14:48:40.021518 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:40.021522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:40.021576 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:40.021625 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:40.022048 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.022903 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.028044 (+ 5141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.028693 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.028827 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:40.028886 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:40.029411 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 14:48:40.029520 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:48:40.030158 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.008331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45337 (local address 127.13.223.65:41669)
0928 14:48:40.018532 (+ 10201us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:40.021218 (+ 2686us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:40.021398 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:40.021772 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:40.021774 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:40.021792 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:40.021908 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:40.021913 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.023075 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.023079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:40.024553 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:40.024570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.029038 (+ 4468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.029043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:40.029155 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:40.029188 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:48:40.029866 (+ 678us) client_negotiation.cc:240] Negotiation successful
0928 14:48:40.029986 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10111,"mutex_wait_us":593,"thread_start_us":114,"threads_started":1}
I20260928 14:48:40.030143 27499 raft_consensus.cc:2781] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:40.030444 27499 raft_consensus.cc:482] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:40.030529 27499 raft_consensus.cc:3037] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:40.032370 27499 raft_consensus.cc:504] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:40.032975 27499 leader_election.cc:288] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [CANDIDATE]: Term 1 election: Requested vote from peers c6b0a29ecd8541bd8951b5fa2ac02a48 (127.13.223.66:45577), d09a957693ac4349b50949d5db0992bb (127.13.223.67:45337)
I20260928 14:48:40.033687 27271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55367ddc094a41e4b46d9367cf9fb74d" candidate_uuid: "ca3eeddd8d324d40b363708480152500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48"
I20260928 14:48:40.033866 27271 raft_consensus.cc:3037] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:40.035452 27271 raft_consensus.cc:2454] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3eeddd8d324d40b363708480152500 in term 1.
I20260928 14:48:40.036186 27082 leader_election.cc:302] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6b0a29ecd8541bd8951b5fa2ac02a48, ca3eeddd8d324d40b363708480152500; no voters:
I20260928 14:48:40.036628 27499 raft_consensus.cc:2781] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:40.036968 27499 raft_consensus.cc:686] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [term 1 LEADER]: Becoming Leader. State: Replica: ca3eeddd8d324d40b363708480152500, State: Running, Role: LEADER
I20260928 14:48:40.037423 27499 consensus_queue.cc:237] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } }
I20260928 14:48:40.041674 26993 catalog_manager.cc:5495] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 reported cstate change: term changed from 0 to 1, leader changed from <none> to ca3eeddd8d324d40b363708480152500 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "ca3eeddd8d324d40b363708480152500" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca3eeddd8d324d40b363708480152500" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 35657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:40.044279 27414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55367ddc094a41e4b46d9367cf9fb74d" candidate_uuid: "ca3eeddd8d324d40b363708480152500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a957693ac4349b50949d5db0992bb" is_pre_election: true
I20260928 14:48:40.044688 27414 raft_consensus.cc:2454] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca3eeddd8d324d40b363708480152500 in term 0.
I20260928 14:48:40.045670 27413 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55367ddc094a41e4b46d9367cf9fb74d" candidate_uuid: "ca3eeddd8d324d40b363708480152500" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d09a957693ac4349b50949d5db0992bb"
I20260928 14:48:40.045846 27413 raft_consensus.cc:3037] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:40.047720 27413 raft_consensus.cc:2454] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca3eeddd8d324d40b363708480152500 in term 1.
I20260928 14:48:40.095944 27488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.089966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.13.223.65:35657)
0928 14:48:40.090104 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:40.090111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:40.090126 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:48:40.090213 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:40.090215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:40.090269 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:40.090389 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:40.090737 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.091568 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.093067 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:40.093555 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:40.093790 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:40.094334 (+ 544us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:40.094949 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:40.095080 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:40.095694 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 14:48:40.095781 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:48:40.103528 27271 raft_consensus.cc:1261] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 1 FOLLOWER]: Refusing update from remote peer ca3eeddd8d324d40b363708480152500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:40.103528 27413 raft_consensus.cc:1261] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 1 FOLLOWER]: Refusing update from remote peer ca3eeddd8d324d40b363708480152500: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:40.104419 27499 consensus_queue.cc:1035] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d09a957693ac4349b50949d5db0992bb" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:40.105116 27499 consensus_queue.cc:1035] T 55367ddc094a41e4b46d9367cf9fb74d P ca3eeddd8d324d40b363708480152500 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6b0a29ecd8541bd8951b5fa2ac02a48" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 45577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:40.112804 27509 mvcc.cc:205] Tried to move back new op lower bound from 7334325944734748672 to 7334325944477179904. Current Snapshot: MvccSnapshot[applied={T|T < 7334325944734748672}]
I20260928 14:48:40.118217 27511 mvcc.cc:205] Tried to move back new op lower bound from 7334325944734748672 to 7334325944477179904. Current Snapshot: MvccSnapshot[applied={T|T < 7334325944734748672}]
I20260928 14:48:40.126510 26994 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:40.126797 26994 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:40.129987 26994 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c168ed36123410891c337f2bdea063f: Sending DeleteTablet for 3 replicas of tablet 55367ddc094a41e4b46d9367cf9fb74d
I20260928 14:48:40.131892 27126 tablet_service.cc:1514] Processing DeleteTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:40 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:38798
I20260928 14:48:40.132581 27394 tablet_service.cc:1514] Processing DeleteTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:40 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:46330
I20260928 14:48:40.133590 27260 tablet_service.cc:1514] Processing DeleteTablet for tablet 55367ddc094a41e4b46d9367cf9fb74d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:40 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:54218
I20260928 14:48:40.137537 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27032
I20260928 14:48:40.139570 27510 mvcc.cc:205] Tried to move back new op lower bound from 7334325944734748672 to 7334325944477179904. Current Snapshot: MvccSnapshot[applied={T|T < 7334325944734748672}]
I20260928 14:48:40.141600 27519 tablet_replica.cc:330] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48: stopping tablet replica
I20260928 14:48:40.142055 27519 raft_consensus.cc:2229] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:40.142412 27518 tablet_replica.cc:330] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: stopping tablet replica
I20260928 14:48:40.142468 27519 raft_consensus.cc:2258] T 55367ddc094a41e4b46d9367cf9fb74d P c6b0a29ecd8541bd8951b5fa2ac02a48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:40.142958 27518 raft_consensus.cc:2229] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:40.143275 27518 raft_consensus.cc:2258] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 14:48:40.146366 27348 connection.cc:535] server connection from 127.13.223.65:41669 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 14:48:40.147220 26978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:35657 (error 108)
I20260928 14:48:40.148943 27518 ts_tablet_manager.cc:1905] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:40.149102 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27196
I20260928 14:48:40.153643 27518 ts_tablet_manager.cc:1918] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:40.153791 27518 log.cc:1196] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/55367ddc094a41e4b46d9367cf9fb74d
I20260928 14:48:40.153950 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.148558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:35657 (local address 127.0.0.1:38816)
0928 14:48:40.153518 (+ 4960us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:40.153638 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:35657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4835,"thread_start_us":236,"threads_started":1}
I20260928 14:48:40.154213 27518 ts_tablet_manager.cc:1939] T 55367ddc094a41e4b46d9367cf9fb74d P d09a957693ac4349b50949d5db0992bb: Deleting consensus metadata
W20260928 14:48:40.154242 26978 catalog_manager.cc:4584] TS ca3eeddd8d324d40b363708480152500 (127.13.223.65:35657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55367ddc094a41e4b46d9367cf9fb74d: Network error: Client connection negotiation failed: client connection to 127.13.223.65:35657: connect: Connection refused (error 111)
I20260928 14:48:40.157670 26979 catalog_manager.cc:4841] TS d09a957693ac4349b50949d5db0992bb (127.13.223.67:45337): tablet 55367ddc094a41e4b46d9367cf9fb74d (table test-table [id=917f284511704b8ea89734a6d3fc6733]) successfully deleted
I20260928 14:48:40.161546 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27329
I20260928 14:48:40.161834 27522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:40.161447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:45577 (local address 127.0.0.1:54226)
0928 14:48:40.161671 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:40.161734 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:45577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260928 14:48:40.169348 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 26961
I20260928 14:48:40.176677 14205 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 14:48:40.523305 14205 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 14:48:40.523478 14205 mini_ranger.cc:66] Stopping Ranger...
I20260928 14:48:40.870007 14205 mini_ranger.cc:68] Stopped Ranger
2026-09-28 14:48:40.870 UTC [26028] LOG: received smart shutdown request
2026-09-28 14:48:40.875 UTC [26028] LOG: background worker "logical replication launcher" (PID 26037) exited with exit code 1
2026-09-28 14:48:40.876 UTC [26032] LOG: shutting down
2026-09-28 14:48:41.138 UTC [26028] LOG: database system is shut down
2026-09-28T14:48:41Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (57505 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:41 dist-test-slave-gkhx krb5kdc[27534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:41 dist-test-slave-gkhx krb5kdc[27534](info): set up 2 sockets
Sep 28 14:48:41 dist-test-slave-gkhx krb5kdc[27534](info): commencing operation
krb5kdc: starting...
W20260928 14:48:42.381879 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:48:42 dist-test-slave-gkhx krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:48:42.540563 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:33871
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:42.677394 27547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:42.677831 27547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:42.677938 27547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:42.683020 27547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:42.683137 27547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:42.683161 27547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:42.683179 27547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:42.683197 27547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:42.691154 27547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:33871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:42.692447 27547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:42.693704 27547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:42.704725 27552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:42.704725 27553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:42.705214 27547 server_base.cc:1034] running on GCE node
W20260928 14:48:42.704725 27555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:42.706166 27547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:42.707828 27547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:42.709059 27547 hybrid_clock.cc:648] HybridClock initialized: now 1790606922708973 us; error 113 us; skew 500 ppm
Sep 28 14:48:42 dist-test-slave-gkhx krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:42.713253 27547 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:42.714743 27547 webserver.cc:460] Webserver started at http://127.13.223.126:39065/ using document root <none> and password file <none>
I20260928 14:48:42.715551 27547 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:42.715866 27547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:42.716179 27547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:42.719671 27547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "d638d7f1eb494eccb62efe7d1e47aadb"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "b0e9804c27f8a01356ba26fe64d90014"
server_key_iv: "ca8487a526ae6baa0b9c7130df4f553e"
server_key_version: "encryptionkey@0"
I20260928 14:48:42.720427 27547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "d638d7f1eb494eccb62efe7d1e47aadb"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "b0e9804c27f8a01356ba26fe64d90014"
server_key_iv: "ca8487a526ae6baa0b9c7130df4f553e"
server_key_version: "encryptionkey@0"
I20260928 14:48:42.725991 27547 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260928 14:48:42.729722 27562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:42.731581 27547 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:42.731815 27547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "d638d7f1eb494eccb62efe7d1e47aadb"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "b0e9804c27f8a01356ba26fe64d90014"
server_key_iv: "ca8487a526ae6baa0b9c7130df4f553e"
server_key_version: "encryptionkey@0"
I20260928 14:48:42.731990 27547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:42.756279 27547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:42.760465 27547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:42.760895 27547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:42.770514 27547 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:33871
I20260928 14:48:42.770887 27614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:33871 every 8 connection(s)
I20260928 14:48:42.771888 27547 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:42.775767 27615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:42.780301 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27547
I20260928 14:48:42.780457 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:48:42.780808 14205 external_mini_cluster.cc:1428] Setting key 9ac3aa660dd28a397c900cd44ef32a3e
I20260928 14:48:42.780678 27615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.125@KRBTEST.COM" created.
Sep 28 14:48:42 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:42.802081 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.784190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33871 (local address 127.0.0.1:53072)
0928 14:48:42.784961 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.785031 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:42.786206 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:42.787485 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:42.787491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:42.788192 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:42.788581 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:42.788599 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:42.790179 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:42.790182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:42.791546 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:42.791561 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:42.792512 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:42.792516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:42.792626 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:42.793754 (+ 1128us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:42.793778 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:42.795996 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:42.798519 (+ 2523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:42.798531 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:42.798544 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:42.798874 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:42.799309 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:42.799314 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:42.799316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:42.799447 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:42.800173 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:42.800180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:42.800329 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 14:48:42.800519 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 14:48:42.800963 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":118,"thread_start_us":175,"threads_started":1}
I20260928 14:48:42.802081 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.785439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53072 (local address 127.13.223.126:33871)
0928 14:48:42.785876 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:42.785890 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:42.785940 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:48:42.786822 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:42.786838 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:42.787185 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:42.787490 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:42.788790 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:42.789766 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:42.791739 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:42.792286 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:42.792534 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:42.796296 (+ 3762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:42.796305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:42.796328 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:42.796357 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:42.798346 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:42.799016 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:42.799023 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:42.799024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:42.799174 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:42.799557 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:42.799561 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:42.799562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:42.800036 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:42.800197 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:42.800594 (+ 397us) server_negotiation.cc:299] Negotiation successful
0928 14:48:42.800907 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":303,"thread_start_us":95,"threads_started":1}
I20260928 14:48:42.805279 27615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:42.806488 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.806038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45350)
0928 14:48:42.806185 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.806341 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:48:42.806839 27563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
I20260928 14:48:42.808264 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.807888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45360)
0928 14:48:42.808121 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.808183 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 14:48:42.808657 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:32995: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
I20260928 14:48:42.815630 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:32995
--webserver_interface=127.13.223.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:42.858111 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } attempt: 1
I20260928 14:48:42.859375 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.858971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45368)
0928 14:48:42.859221 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.859290 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260928 14:48:42.860383 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.860090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45372)
0928 14:48:42.860258 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.860314 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 14:48:42.860738 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:32995: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
I20260928 14:48:42.929018 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } attempt: 2
I20260928 14:48:42.930321 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.929785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45380)
0928 14:48:42.930032 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.930121 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260928 14:48:42.931579 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:42.931269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45384)
0928 14:48:42.931423 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:42.931494 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260928 14:48:42.931885 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:32995: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
W20260928 14:48:42.946074 27622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:42.946448 27622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:42.946494 27622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:42.951089 27622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:42.951179 27622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:42.951200 27622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:42.951217 27622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:42.951233 27622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:42.957923 27622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:32995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:42.959441 27622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:42.961087 27622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:42.972664 27630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:42.972700 27627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:42.973321 27628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:42.973421 27622 server_base.cc:1034] running on GCE node
I20260928 14:48:42.974436 27622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:42.976210 27622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:42.977523 27622 hybrid_clock.cc:648] HybridClock initialized: now 1790606922977458 us; error 84 us; skew 500 ppm
Sep 28 14:48:42 dist-test-slave-gkhx krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:42.981864 27622 init.cc:377] Logged in from keytab as kudu/127.13.223.125@KRBTEST.COM (short username kudu)
I20260928 14:48:42.983438 27622 webserver.cc:460] Webserver started at http://127.13.223.125:46843/ using document root <none> and password file <none>
I20260928 14:48:42.984520 27622 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:42.984616 27622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:42.984822 27622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:42.987914 27622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data/instance:
uuid: "c0a5aeec8cea41f1bc318645524e480d"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "e6df3a2d4704d0b0e43860b8b61839e5"
server_key_iv: "a8c83031a11d8ad0231548c72d54c36e"
server_key_version: "encryptionkey@0"
I20260928 14:48:42.988670 27622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal/instance:
uuid: "c0a5aeec8cea41f1bc318645524e480d"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "e6df3a2d4704d0b0e43860b8b61839e5"
server_key_iv: "a8c83031a11d8ad0231548c72d54c36e"
server_key_version: "encryptionkey@0"
I20260928 14:48:42.997011 27622 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260928 14:48:43.001266 27637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:43.003436 27622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 14:48:43.003618 27622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "c0a5aeec8cea41f1bc318645524e480d"
format_stamp: "Formatted at 2026-09-28 14:48:42 on dist-test-slave-gkhx"
server_key: "e6df3a2d4704d0b0e43860b8b61839e5"
server_key_iv: "a8c83031a11d8ad0231548c72d54c36e"
server_key_version: "encryptionkey@0"
I20260928 14:48:43.003757 27622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:43.019538 27622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:43.023137 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } attempt: 3
I20260928 14:48:43.023804 27622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:43.024050 27622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:43.024188 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.023768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45398)
0928 14:48:43.024004 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.024071 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260928 14:48:43.025205 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.024889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45404)
0928 14:48:43.025051 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.025115 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 14:48:43.025441 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:32995: Network error: Client connection negotiation failed: client connection to 127.13.223.125:32995: connect: Connection refused (error 111)
I20260928 14:48:43.032819 27622 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.125:32995
I20260928 14:48:43.033213 27689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.125:32995 every 8 connection(s)
I20260928 14:48:43.034071 27622 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
I20260928 14:48:43.035246 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27622
I20260928 14:48:43.035475 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-1/wal/instance
I20260928 14:48:43.035969 14205 external_mini_cluster.cc:1428] Setting key ccf510076d2efa9ace124a929c3213cf
I20260928 14:48:43.038851 27690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
WARNING: no policy specified for kudu/127.13.223.124@KRBTEST.COM; defaulting to no policy
I20260928 14:48:43.045045 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.124@KRBTEST.COM" created.
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:43.067763 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.047995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.13.223.126:33871)
0928 14:48:43.048213 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.048217 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.049632 (+ 1415us) server_negotiation.cc:407] Connection header received
0928 14:48:43.051681 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.051684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.051746 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.051834 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.053989 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.055126 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.056867 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.057150 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.057305 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.061082 (+ 3777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.061086 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.061088 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.061122 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.064349 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.065407 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.065412 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.065414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.065523 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.066149 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.066153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.066155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.066394 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.066539 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.067357 (+ 818us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.067552 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260928 14:48:43.068986 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.048612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33871 (local address 127.0.0.1:53080)
0928 14:48:43.049406 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.049459 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.051295 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.052111 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.052125 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.052890 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.053748 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.053772 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.055429 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.055433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.056659 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.056684 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.057382 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.057387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.057612 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.058887 (+ 1275us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.058910 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.060807 (+ 1897us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.064611 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.064619 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.064640 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.065245 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.065716 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.065719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.065721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.065921 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.066609 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.066623 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.067020 (+ 397us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.067277 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.067718 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":172,"threads_started":1}
I20260928 14:48:43.071895 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
I20260928 14:48:43.085060 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.072868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45410)
0928 14:48:43.073082 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.073103 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.073211 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.074034 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.074037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.074280 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.074370 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.074376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.075578 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.075583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.076157 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.076168 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.077478 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.077481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.077585 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.078521 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.078535 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.080263 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.082850 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.082856 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.082859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.083103 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.083486 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.083490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.083492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.083841 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.084535 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.084538 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.084637 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.084751 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.084913 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260928 14:48:43.085777 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.073061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45410 (local address 127.13.223.125:32995)
0928 14:48:43.073551 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.073562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.073586 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:48:43.073634 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.073641 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.073850 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.074053 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.074501 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.075422 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.076308 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.077200 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.077477 (+ 277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.080469 (+ 2992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.080486 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.080502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.080530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.082683 (+ 2153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.083250 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.083259 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.083261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.083363 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.084008 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.084015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.084017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.084410 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.084587 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.085173 (+ 586us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.085366 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":231,"threads_started":1}
I20260928 14:48:43.085937 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:46201
--webserver_interface=127.13.223.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:43.089722 27690 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:43.091032 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.090529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54028)
0928 14:48:43.090756 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.090897 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260928 14:48:43.091382 27638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.092974 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.092328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54032)
0928 14:48:43.092674 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.092877 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260928 14:48:43.093343 27690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.142822 27690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 1
I20260928 14:48:43.143862 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.143482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54036)
0928 14:48:43.143704 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.143774 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260928 14:48:43.144894 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.144543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54046)
0928 14:48:43.144748 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.144819 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260928 14:48:43.145156 27690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.168672 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } attempt: 4
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:48:43.183302 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.169374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45424)
0928 14:48:43.169586 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.169609 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.169741 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.170115 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.170118 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.170787 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.171157 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.171189 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.172616 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.172621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.173289 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.173305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.174045 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.174048 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.174175 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.174836 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.174849 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.176311 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.180804 (+ 4493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.180809 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.180871 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.181247 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.181865 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.181869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.181871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.181994 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.182633 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.182636 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.182768 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.182894 (+ 126us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.183106 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20260928 14:48:43.183302 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.169651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45424 (local address 127.13.223.125:32995)
0928 14:48:43.169877 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.169882 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.169898 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:48:43.169943 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.169946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.169991 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.170089 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.171483 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.172129 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.173519 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.173846 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.174038 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.176543 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.176546 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.176548 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.176586 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.180554 (+ 3968us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.181417 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.181421 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.181484 (+ 63us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.181606 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.182114 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.182117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.182119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.182476 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.182650 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.182972 (+ 322us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.183116 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260928 14:48:43.184338 27615 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:43.185268 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.184928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54054)
0928 14:48:43.185102 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.185180 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 14:48:43.186084 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.185830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54064)
0928 14:48:43.185954 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.186014 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:48:43.186226 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.213382 27690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 2
I20260928 14:48:43.214776 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.214108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54068)
0928 14:48:43.214368 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.214649 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260928 14:48:43.216306 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.215868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54070)
0928 14:48:43.216037 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.216112 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 14:48:43.216569 27690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
W20260928 14:48:43.234093 27696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:43.234452 27696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:43.234498 27696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:43.240279 27696 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:43.240404 27696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:43.240437 27696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:43.240468 27696 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:43.240497 27696 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:43.245453 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 1
I20260928 14:48:43.246567 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.246073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54080)
0928 14:48:43.246364 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.246439 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260928 14:48:43.247558 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.247231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54082)
0928 14:48:43.247405 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.247479 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 14:48:43.247776 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.248875 27696 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:33871,127.13.223.125:32995,127.13.223.124:46201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:43.250200 27696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:43.251495 27696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:43.262455 27705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:43.262437 27703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:43.262435 27702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:43.262941 27696 server_base.cc:1034] running on GCE node
I20260928 14:48:43.263870 27696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:43.265434 27696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:43.268879 27696 hybrid_clock.cc:648] HybridClock initialized: now 1790606923268814 us; error 83 us; skew 500 ppm
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:43.272712 27696 init.cc:377] Logged in from keytab as kudu/127.13.223.124@KRBTEST.COM (short username kudu)
I20260928 14:48:43.274753 27696 webserver.cc:460] Webserver started at http://127.13.223.124:41853/ using document root <none> and password file <none>
I20260928 14:48:43.276458 27696 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:43.276654 27696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:43.276954 27696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:43.279976 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data/instance:
uuid: "f6da0cc943204c86baa1c3509e9b8cdf"
format_stamp: "Formatted at 2026-09-28 14:48:43 on dist-test-slave-gkhx"
server_key: "5c285d770751e9aa4c3b6878735e44fb"
server_key_iv: "7030fd38830e1cede63cca57fae495f5"
server_key_version: "encryptionkey@0"
I20260928 14:48:43.281070 27696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal/instance:
uuid: "f6da0cc943204c86baa1c3509e9b8cdf"
format_stamp: "Formatted at 2026-09-28 14:48:43 on dist-test-slave-gkhx"
server_key: "5c285d770751e9aa4c3b6878735e44fb"
server_key_iv: "7030fd38830e1cede63cca57fae495f5"
server_key_version: "encryptionkey@0"
I20260928 14:48:43.287057 27696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260928 14:48:43.291457 27712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:43.293536 27696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:48:43.293763 27696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "f6da0cc943204c86baa1c3509e9b8cdf"
format_stamp: "Formatted at 2026-09-28 14:48:43 on dist-test-slave-gkhx"
server_key: "5c285d770751e9aa4c3b6878735e44fb"
server_key_iv: "7030fd38830e1cede63cca57fae495f5"
server_key_version: "encryptionkey@0"
I20260928 14:48:43.293932 27696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:43.307837 27690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 3
I20260928 14:48:43.308888 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.308507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54084)
0928 14:48:43.308716 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.308788 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260928 14:48:43.309937 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.309568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54088)
0928 14:48:43.309807 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.309867 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260928 14:48:43.310094 27690 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.315004 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 2
I20260928 14:48:43.316180 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.315781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54098)
0928 14:48:43.316003 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.316080 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 14:48:43.317186 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.316873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54100)
0928 14:48:43.317031 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.317114 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 14:48:43.317440 27615 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:46201: Network error: Client connection negotiation failed: client connection to 127.13.223.124:46201: connect: Connection refused (error 111)
I20260928 14:48:43.360244 27696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:43.364928 27696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:43.365211 27696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:43.374771 27696 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.124:46201
I20260928 14:48:43.375082 27764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.124:46201 every 8 connection(s)
I20260928 14:48:43.376073 27696 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
I20260928 14:48:43.378403 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27696
I20260928 14:48:43.378638 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790606739990168-14205-0/minicluster-data/master-2/wal/instance
I20260928 14:48:43.378957 14205 external_mini_cluster.cc:1428] Setting key 7602775d2d7bc3806611425259746ed1
I20260928 14:48:43.379841 27765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:43.385264 27765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:43.399328 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.380663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54102 (local address 127.13.223.124:46201)
0928 14:48:43.381190 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.381214 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.381511 (+ 297us) server_negotiation.cc:407] Connection header received
0928 14:48:43.382439 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.382460 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.382947 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.383386 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.384317 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.385444 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.386486 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.387195 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.387396 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.391520 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.391526 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.391539 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.391581 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.394219 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.395009 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.395015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.395024 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.395189 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.395714 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.395718 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.395719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.396398 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.396665 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.397152 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.397636 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":168,"threads_started":1}
I20260928 14:48:43.401934 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.387578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53094 (local address 127.13.223.126:33871)
0928 14:48:43.387757 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.387762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.388212 (+ 450us) server_negotiation.cc:407] Connection header received
0928 14:48:43.388560 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.388563 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.388613 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.388688 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.390029 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.391064 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.392479 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.392932 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.393086 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.396217 (+ 3131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.396222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.396225 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.396259 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.398845 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.399814 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.399819 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.399820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.399931 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.400530 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.400536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.400538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.400812 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.400901 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.401497 (+ 596us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.401773 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:48:43.402206 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.387488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:33871 (local address 127.0.0.1:53094)
0928 14:48:43.388041 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.388092 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.388336 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.388796 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.388804 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.389416 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.389845 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.389864 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.391247 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.391251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.392323 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.392336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.393092 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.393096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.393264 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.393893 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.393917 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.396009 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.399097 (+ 3088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.399107 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.399112 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.399699 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.400292 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.400295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.400296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.400401 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.400935 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.400941 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.401202 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.401382 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.401620 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":159,"threads_started":1}
I20260928 14:48:43.403702 27765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:48:43.415596 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.404536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.13.223.125:32995)
0928 14:48:43.404665 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.404670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.404689 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:48:43.404730 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.404733 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.404772 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.404845 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.405436 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.406122 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.407253 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.407677 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.407814 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.410796 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.410802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.410806 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.410846 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.412910 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.413779 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.413783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.413785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.413920 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.414373 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.414376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.414378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.414772 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.414880 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.415274 (+ 394us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.415429 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:48:43.415596 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.402830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.13.223.126:33871)
0928 14:48:43.402937 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.402943 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.402958 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:48:43.403015 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.403017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.403098 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.403189 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.404096 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.404896 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.405987 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.406305 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.406450 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.410866 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.410870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.410873 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.410905 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.413062 (+ 2157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.413795 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.413800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.413802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.413920 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.414371 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.414376 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.414377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.414737 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.415023 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.415274 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.415427 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 14:48:43.415653 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.404356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:32995 (local address 127.0.0.1:45436)
0928 14:48:43.404533 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.404547 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.404627 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.404893 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.404895 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.405183 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.405298 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.405305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.406302 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.406305 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.407041 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.407053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.407986 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.407992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.408188 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.408744 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.408765 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.410401 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.413213 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.413218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.413220 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.413565 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.414064 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.414067 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.414070 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.414206 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.414908 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.414912 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.415037 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.415167 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.415346 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260928 14:48:43.417140 27765 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:48:43.417594 27615 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 3
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606923, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:48:43.429144 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.417229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45444 (local address 127.13.223.125:32995)
0928 14:48:43.417352 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.417356 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.417559 (+ 203us) server_negotiation.cc:407] Connection header received
0928 14:48:43.417862 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.417864 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.417908 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.418003 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.418958 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.419708 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.420534 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.420873 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.420986 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.424534 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.424539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.424542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.424578 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.426792 (+ 2214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.427448 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.427452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.427454 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.427553 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.427962 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.427966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.427968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.428317 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.428615 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.428783 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.428931 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:48:43.429533 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.418297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54126)
0928 14:48:43.418403 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.418419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.418501 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.419099 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.419102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.419303 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.419406 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.419413 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.420588 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.420591 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.421304 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.421314 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.421964 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.421968 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.422059 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.422592 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.422607 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.424790 (+ 2183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.427454 (+ 2664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.427460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.427466 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.427767 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.428112 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.428118 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.428120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.428227 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.428989 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.428993 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.429100 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.429206 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.429374 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 14:48:43.429965 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.417958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54116)
0928 14:48:43.418480 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.418492 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.418933 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.419282 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.419284 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.419422 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.419531 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.419537 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.420863 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.420865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.421649 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.421660 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.422431 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.422434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.422570 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.423061 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.423083 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.425298 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.427886 (+ 2588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.427891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.427895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.428225 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.428546 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.428549 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.428551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.428790 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.429351 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.429354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.429439 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.429561 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.429765 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"spinlock_wait_cycles":7040}
I20260928 14:48:43.430007 27768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.418028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.13.223.124:46201)
0928 14:48:43.418159 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.418165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.418888 (+ 723us) server_negotiation.cc:407] Connection header received
0928 14:48:43.419010 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.419016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.419060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.419124 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.419642 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.420777 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.421762 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.422177 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.422302 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.425459 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.425465 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.425469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.425495 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.427746 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.428344 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.428350 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.428355 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.428440 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.428936 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.428939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.428941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.429201 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.429280 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.429612 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.429866 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":134,"server-negotiator.queue_time_us":38,"spinlock_wait_cycles":4864}
I20260928 14:48:43.430625 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.418363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54126 (local address 127.13.223.124:46201)
0928 14:48:43.418873 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.418883 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.418897 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:48:43.418937 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.418942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.418991 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.419079 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.419539 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.420444 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.421457 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.421841 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.421959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.425041 (+ 3082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.425049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.425052 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.425091 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.427232 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.427890 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.427893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.427895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.427987 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.428543 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.428545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.428547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.428859 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.429836 (+ 977us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.429927 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.430366 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"spinlock_wait_cycles":1408,"thread_start_us":161,"threads_started":1}
I20260928 14:48:43.433774 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:43.434043 27765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: Bootstrap starting.
I20260928 14:48:43.434043 27615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: Bootstrap starting.
I20260928 14:48:43.437624 27765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:43.437644 27615 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:43.438872 27615 log.cc:824] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:43.438872 27765 log.cc:824] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:43.441627 27765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: No bootstrap required, opened a new log
I20260928 14:48:43.449440 27615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: No bootstrap required, opened a new log
I20260928 14:48:43.450905 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.436990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54130 (local address 127.13.223.124:46201)
0928 14:48:43.437128 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.437133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.438268 (+ 1135us) server_negotiation.cc:407] Connection header received
0928 14:48:43.438490 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.438494 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.438580 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.438652 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.439959 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.440865 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.442286 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.442709 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.442806 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.445571 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.445575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.445578 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.445627 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.448067 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.448678 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.448686 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.448687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.449035 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.449495 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.449501 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.449505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.449786 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.449923 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.450518 (+ 595us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.450737 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:48:43.450920 27695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.437297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.13.223.125:32995)
0928 14:48:43.437442 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.437447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.437749 (+ 302us) server_negotiation.cc:407] Connection header received
0928 14:48:43.437937 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.437941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.438026 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.438121 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.439427 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.440619 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.442203 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.442638 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.442732 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.445616 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.445621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.445624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.445662 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.448109 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.448993 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.448999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.449004 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.449146 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.449551 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.449557 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.449560 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.449905 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.450070 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.450533 (+ 463us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.450717 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:48:43.453718 27690 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } attempt: 4
I20260928 14:48:43.456768 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.436593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.13.223.126:33871)
0928 14:48:43.436755 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.436763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.442178 (+ 5415us) server_negotiation.cc:407] Connection header received
0928 14:48:43.442398 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.442401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.442458 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.442516 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.443753 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.444327 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.445577 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.446043 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.446133 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.448379 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.448388 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.448395 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.448438 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.450672 (+ 2234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.451367 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.451372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.451374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.451481 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.452901 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.452905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.452906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.453271 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.453524 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.456223 (+ 2699us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.456405 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 14:48:43.459403 27615 raft_consensus.cc:348] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.459923 27615 raft_consensus.cc:374] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:43.459308 27765 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.460041 27615 raft_consensus.cc:729] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d638d7f1eb494eccb62efe7d1e47aadb, State: Initialized, Role: FOLLOWER
I20260928 14:48:43.460104 27765 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:43.460227 27765 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6da0cc943204c86baa1c3509e9b8cdf, State: Initialized, Role: FOLLOWER
I20260928 14:48:43.460776 27615 consensus_queue.cc:260] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.460800 27765 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.462757 27615 sys_catalog.cc:564] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 14:48:43 dist-test-slave-gkhx krb5kdc[27534](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606922, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:48:43.463107 27765 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:43.466185 27797 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 14:48:43.467401 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.468153 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.467824 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.468390 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.471058 27693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.454462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:46201 (local address 127.0.0.1:54144)
0928 14:48:43.454937 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:43.454956 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:43.455056 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:43.456397 (+ 1341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:43.456400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:43.456743 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:43.456895 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.456905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.458392 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.458397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.459798 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:43.459816 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.460798 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.460802 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:43.460906 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.461554 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:43.461571 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:43.464916 (+ 3345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:43.468565 (+ 3649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.468570 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.468573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.469016 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.469486 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:43.469490 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:43.469492 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:43.469626 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:43.470230 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:43.470234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:43.470359 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 14:48:43.470454 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:48:43.470849 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369}
I20260928 14:48:43.472323 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:43.455588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54144 (local address 127.13.223.124:46201)
0928 14:48:43.455979 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:43.455986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:43.456004 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:48:43.456052 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:43.456058 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:43.456137 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:43.456239 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:43.457090 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.458215 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.460016 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:43.460470 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:43.460594 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:43.465144 (+ 4550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:43.465149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:43.465152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:43.465185 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:43.468377 (+ 3192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.469219 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.469226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.469232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.469353 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:43.469770 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:43.469774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:43.469776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:43.470075 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:43.471771 (+ 1696us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:43.471991 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:48:43.472163 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
W20260928 14:48:43.473328 27803 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:43.473428 27803 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:48:43.470988 27547 master_runner.cc:186] Error getting master registration for 127.13.223.125:32995: OK, instance_id { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" instance_seqno: 1790606923030354 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:48:43.476341 27547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:48:43.476960 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: Bootstrap starting.
I20260928 14:48:43.480626 27781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:43.480901 27781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.481127 27690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:43.482465 27690 log.cc:824] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:43.483506 27781 leader_election.cc:288] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0a5aeec8cea41f1bc318645524e480d (127.13.223.125:32995), f6da0cc943204c86baa1c3509e9b8cdf (127.13.223.124:46201)
I20260928 14:48:43.481005 27696 master_runner.cc:186] Error getting master registration for 127.13.223.125:32995: OK, instance_id { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" instance_seqno: 1790606923030354 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:48:43.485476 27696 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:48:43.485926 27739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" is_pre_election: true
I20260928 14:48:43.486681 27739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d638d7f1eb494eccb62efe7d1e47aadb in term 0.
I20260928 14:48:43.487713 27563 leader_election.cc:302] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d638d7f1eb494eccb62efe7d1e47aadb, f6da0cc943204c86baa1c3509e9b8cdf; no voters:
I20260928 14:48:43.489200 27781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:43.489426 27781 raft_consensus.cc:482] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:43.489519 27781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:43.490491 27690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: No bootstrap required, opened a new log
I20260928 14:48:43.491328 27781 raft_consensus.cc:504] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.491838 27781 leader_election.cc:288] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [CANDIDATE]: Term 1 election: Requested vote from peers c0a5aeec8cea41f1bc318645524e480d (127.13.223.125:32995), f6da0cc943204c86baa1c3509e9b8cdf (127.13.223.124:46201)
I20260928 14:48:43.492880 27739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6da0cc943204c86baa1c3509e9b8cdf"
I20260928 14:48:43.493073 27739 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:43.495415 27739 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d638d7f1eb494eccb62efe7d1e47aadb in term 1.
I20260928 14:48:43.496291 27690 raft_consensus.cc:348] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.496677 27690 raft_consensus.cc:374] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:43.496788 27690 raft_consensus.cc:729] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0a5aeec8cea41f1bc318645524e480d, State: Initialized, Role: FOLLOWER
I20260928 14:48:43.497625 27690 consensus_queue.cc:260] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.498399 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.498709 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.499547 27563 leader_election.cc:302] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d638d7f1eb494eccb62efe7d1e47aadb, f6da0cc943204c86baa1c3509e9b8cdf; no voters:
I20260928 14:48:43.499672 27690 sys_catalog.cc:564] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:43.499820 27781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:43.499830 27664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a5aeec8cea41f1bc318645524e480d" is_pre_election: true
I20260928 14:48:43.500142 27781 raft_consensus.cc:686] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [term 1 LEADER]: Becoming Leader. State: Replica: d638d7f1eb494eccb62efe7d1e47aadb, State: Running, Role: LEADER
I20260928 14:48:43.500226 27664 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d638d7f1eb494eccb62efe7d1e47aadb in term 0.
I20260928 14:48:43.500555 27781 consensus_queue.cc:237] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } }
I20260928 14:48:43.502477 27663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0a5aeec8cea41f1bc318645524e480d"
I20260928 14:48:43.503073 27663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:43.502910 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d638d7f1eb494eccb62efe7d1e47aadb. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.505756 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:43.505848 27663 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d638d7f1eb494eccb62efe7d1e47aadb in term 1.
W20260928 14:48:43.509444 27818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:48:43.509582 27818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 14:48:43.513535 27568 tablet.cc:2369] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:43.520859 27821 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:43.525159 27821 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260928 14:48:43.526618 27643 tablet.cc:2369] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:43.532487 27739 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [term 1 FOLLOWER]: Refusing update from remote peer d638d7f1eb494eccb62efe7d1e47aadb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:43.533262 27663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [term 1 FOLLOWER]: Refusing update from remote peer d638d7f1eb494eccb62efe7d1e47aadb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:43.534698 27807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:43.536726 27807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:43.540485 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d638d7f1eb494eccb62efe7d1e47aadb. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.540674 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.542634 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader d638d7f1eb494eccb62efe7d1e47aadb. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.542814 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.554155 27807 sys_catalog.cc:455] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.554350 27807 sys_catalog.cc:458] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:43.554489 27823 mvcc.cc:205] Tried to move back new op lower bound from 7334325958779838464 to 7334325958662619136. Current Snapshot: MvccSnapshot[applied={T|T < 7334325958779838464}]
I20260928 14:48:43.555321 27781 sys_catalog.cc:455] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.555476 27781 sys_catalog.cc:458] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:43.556691 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.556833 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.558418 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d638d7f1eb494eccb62efe7d1e47aadb" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 33871 } } peers { permanent_uuid: "c0a5aeec8cea41f1bc318645524e480d" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 32995 } } peers { permanent_uuid: "f6da0cc943204c86baa1c3509e9b8cdf" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 46201 } } }
I20260928 14:48:43.558604 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:48:43.559255 27826 mvcc.cc:205] Tried to move back new op lower bound from 7334325958779838464 to 7334325958662619136. Current Snapshot: MvccSnapshot[applied={T|T < 7334325958779838464}]
I20260928 14:48:43.560416 27821 catalog_manager.cc:1347] Generated new cluster ID: cc63430ed35c493dad9cacd49ecdfdae
I20260928 14:48:43.560496 27821 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:43.626391 27821 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:43.628173 27821 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:43.656637 27821 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d638d7f1eb494eccb62efe7d1e47aadb: Generated new TSK 0
I20260928 14:48:43.657802 27821 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:44.474939 27803 catalog_manager.cc:1259] Loaded cluster ID: cc63430ed35c493dad9cacd49ecdfdae
I20260928 14:48:44.475102 27803 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: loading cluster ID for follower catalog manager: success
I20260928 14:48:44.477200 27803 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: acquiring CA information for follower catalog manager: success
I20260928 14:48:44.479017 27803 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f6da0cc943204c86baa1c3509e9b8cdf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:48:44.511662 27818 catalog_manager.cc:1259] Loaded cluster ID: cc63430ed35c493dad9cacd49ecdfdae
I20260928 14:48:44.511782 27818 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: loading cluster ID for follower catalog manager: success
I20260928 14:48:44.513870 27818 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: acquiring CA information for follower catalog manager: success
I20260928 14:48:44.515532 27818 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c0a5aeec8cea41f1bc318645524e480d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:48:44.741427 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27547
I20260928 14:48:44.748694 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27622
I20260928 14:48:44.755870 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27696
2026-09-28T14:48:44Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3494 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:44 dist-test-slave-gkhx krb5kdc[27839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:44 dist-test-slave-gkhx krb5kdc[27839](info): set up 2 sockets
Sep 28 14:48:44 dist-test-slave-gkhx krb5kdc[27839](info): commencing operation
krb5kdc: starting...
W20260928 14:48:45.893051 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.112s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:48:45 dist-test-slave-gkhx krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:48:46.055472 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:32777
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:36357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:32777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:46.186719 27852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:46.187059 27852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:46.187104 27852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:46.191381 27852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:46.191459 27852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:46.191479 27852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:46.191496 27852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:46.191514 27852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:46.197613 27852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:36357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:32777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:32777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:46.198817 27852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:46.199880 27852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:46.209301 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:46.209309 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:46.209309 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:46.209681 27852 server_base.cc:1034] running on GCE node
I20260928 14:48:46.210350 27852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:46.211692 27852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:46.212924 27852 hybrid_clock.cc:648] HybridClock initialized: now 1790606926212890 us; error 51 us; skew 500 ppm
Sep 28 14:48:46 dist-test-slave-gkhx krb5kdc[27839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606926, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:46.216209 27852 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:46.217355 27852 webserver.cc:460] Webserver started at http://127.13.223.126:45343/ using document root <none> and password file <none>
I20260928 14:48:46.218053 27852 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:46.218175 27852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:46.218438 27852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:46.220918 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0"
format_stamp: "Formatted at 2026-09-28 14:48:46 on dist-test-slave-gkhx"
server_key: "3d928fd120b1b664f4f3adb41ec65748"
server_key_iv: "1a2681be5c60bba6b1e0d906fc8677f9"
server_key_version: "encryptionkey@0"
I20260928 14:48:46.221649 27852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0"
format_stamp: "Formatted at 2026-09-28 14:48:46 on dist-test-slave-gkhx"
server_key: "3d928fd120b1b664f4f3adb41ec65748"
server_key_iv: "1a2681be5c60bba6b1e0d906fc8677f9"
server_key_version: "encryptionkey@0"
I20260928 14:48:46.226943 27852 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260928 14:48:46.230191 27867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:46.232009 27852 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:46.232290 27852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0"
format_stamp: "Formatted at 2026-09-28 14:48:46 on dist-test-slave-gkhx"
server_key: "3d928fd120b1b664f4f3adb41ec65748"
server_key_iv: "1a2681be5c60bba6b1e0d906fc8677f9"
server_key_version: "encryptionkey@0"
I20260928 14:48:46.232453 27852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:46.256980 27852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:46.260993 27852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:46.261303 27852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:46.269918 27852 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:32777
I20260928 14:48:46.270051 27919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:32777 every 8 connection(s)
I20260928 14:48:46.271548 27852 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:46.274986 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27852
I20260928 14:48:46.275146 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:48:46.275499 14205 external_mini_cluster.cc:1428] Setting key 17b8a5fb0a9b9c4eded9879e34ec7d62
I20260928 14:48:46.276700 27920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:46.284541 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0: Bootstrap starting.
Sep 28 14:48:46 dist-test-slave-gkhx krb5kdc[27839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:46.287647 27920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:46.288571 27920 log.cc:824] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:46.291393 27920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0: No bootstrap required, opened a new log
I20260928 14:48:46.294771 27923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:46.277543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.13.223.126:32777)
0928 14:48:46.278282 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:46.278309 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:46.278364 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:48:46.279506 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:46.279537 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:46.280034 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:46.280449 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:46.281445 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:46.282962 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:46.283950 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:46.284539 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:46.285246 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:46.288337 (+ 3091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:46.288349 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:46.288367 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:46.288404 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:46.290797 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:46.291391 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:46.291398 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:46.291406 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:46.291565 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:46.291964 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:46.291968 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:46.291970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:46.292563 (+ 593us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:46.292779 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:46.293131 (+ 352us) server_negotiation.cc:299] Negotiation successful
0928 14:48:46.293487 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":180,"threads_started":1}
I20260928 14:48:46.296725 27920 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } }
I20260928 14:48:46.297086 27920 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:46.297153 27920 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdafbfd4e7614f27a603ce0dcfbe4df0, State: Initialized, Role: FOLLOWER
I20260928 14:48:46.297732 27920 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } }
I20260928 14:48:46.297940 27920 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:46.298015 27920 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:46.298131 27920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:46.298382 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:48:46.299589 27920 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } }
I20260928 14:48:46.300058 27920 leader_election.cc:302] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdafbfd4e7614f27a603ce0dcfbe4df0; no voters:
I20260928 14:48:46.300448 27920 leader_election.cc:288] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:46.300575 27925 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:46.300863 27925 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [term 1 LEADER]: Becoming Leader. State: Replica: bdafbfd4e7614f27a603ce0dcfbe4df0, State: Running, Role: LEADER
I20260928 14:48:46.301194 27925 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } }
I20260928 14:48:46.301990 27920 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:46.303210 27926 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } } }
I20260928 14:48:46.303346 27926 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:46.303579 27927 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdafbfd4e7614f27a603ce0dcfbe4df0. Latest consensus state: current_term: 1 leader_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdafbfd4e7614f27a603ce0dcfbe4df0" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 32777 } } }
I20260928 14:48:46.303664 27927 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:46.304250 27934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:46.307041 27934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:46.312809 27934 catalog_manager.cc:1347] Generated new cluster ID: 6276a1d4f08e41d69d6430a2e079c1df
I20260928 14:48:46.312904 27934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:46.328979 27934 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:46.330471 27934 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:46.338249 27934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdafbfd4e7614f27a603ce0dcfbe4df0: Generated new TSK 0
I20260928 14:48:46.339134 27934 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:48:46.355439 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27852
2026-09-28T14:48:46Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1595 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:48:46 dist-test-slave-gkhx krb5kdc[27947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:48:46 dist-test-slave-gkhx krb5kdc[27947](info): set up 2 sockets
Sep 28 14:48:46 dist-test-slave-gkhx krb5kdc[27947](info): commencing operation
krb5kdc: starting...
W20260928 14:48:47.460961 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:48:47 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:48:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:48:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:48:47.607568 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:35385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:47.749985 27960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:47.750387 27960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:47.750460 27960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:47.754942 27960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:47.755247 27960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:47.755357 27960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:47.755424 27960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:47.755489 27960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:47.763113 27960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:35385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:47.764545 27960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:47.765820 27960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:47.777274 27965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:47.777346 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:47.777612 27960 server_base.cc:1034] running on GCE node
W20260928 14:48:47.777346 27966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:47.778527 27960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:47.779850 27960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:47.781118 27960 hybrid_clock.cc:648] HybridClock initialized: now 1790606927781085 us; error 59 us; skew 500 ppm
Sep 28 14:48:47 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606927, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:47.784699 27960 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:47.785873 27960 webserver.cc:460] Webserver started at http://127.13.223.126:43549/ using document root <none> and password file <none>
I20260928 14:48:47.786525 27960 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:47.786702 27960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:47.786964 27960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:47.789310 27960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "c4121fe0d80648a9bfe7ed669dc28f01"
format_stamp: "Formatted at 2026-09-28 14:48:47 on dist-test-slave-gkhx"
server_key: "2e8aec4be54d39da0ce725ebdd7a85f0"
server_key_iv: "ee5ffb0c23fbe5de459ec82c507df686"
server_key_version: "encryptionkey@0"
I20260928 14:48:47.789978 27960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "c4121fe0d80648a9bfe7ed669dc28f01"
format_stamp: "Formatted at 2026-09-28 14:48:47 on dist-test-slave-gkhx"
server_key: "2e8aec4be54d39da0ce725ebdd7a85f0"
server_key_iv: "ee5ffb0c23fbe5de459ec82c507df686"
server_key_version: "encryptionkey@0"
I20260928 14:48:47.794947 27960 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260928 14:48:47.798246 27975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:47.800290 27960 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:47.800601 27960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c4121fe0d80648a9bfe7ed669dc28f01"
format_stamp: "Formatted at 2026-09-28 14:48:47 on dist-test-slave-gkhx"
server_key: "2e8aec4be54d39da0ce725ebdd7a85f0"
server_key_iv: "ee5ffb0c23fbe5de459ec82c507df686"
server_key_version: "encryptionkey@0"
I20260928 14:48:47.800802 27960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:47.821290 27960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:47.825589 27960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:47.825863 27960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:47.833995 27960 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:35385
I20260928 14:48:47.834353 28027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:35385 every 8 connection(s)
I20260928 14:48:47.835211 27960 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:47.836757 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 27960
I20260928 14:48:47.836937 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:48:47.837352 14205 external_mini_cluster.cc:1428] Setting key 04a0c661cf6713f026cd0fc1f750afda
I20260928 14:48:47.838963 28028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:47.846735 28028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap starting.
Sep 28 14:48:47 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:47.849735 28028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:47.850819 28028 log.cc:824] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:47.853535 28028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: No bootstrap required, opened a new log
I20260928 14:48:47.855890 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:47.839210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57128 (local address 127.13.223.126:35385)
0928 14:48:47.839703 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:47.839725 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:47.839769 (+ 44us) server_negotiation.cc:407] Connection header received
0928 14:48:47.840660 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:47.840686 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:47.841159 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:47.841593 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:47.842503 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:47.844115 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:47.845311 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:47.846095 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:47.846295 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:47.849658 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:47.849667 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:47.849688 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:47.849733 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:47.851979 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:47.852642 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:47.852649 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:47.852655 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:47.852800 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:47.853176 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:47.853180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:47.853182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:47.853747 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:47.853956 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:47.854238 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 14:48:47.854602 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":143,"threads_started":1}
I20260928 14:48:47.857776 28028 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:47.858093 28028 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:47.858155 28028 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Initialized, Role: FOLLOWER
I20260928 14:48:47.858706 28028 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:47.858907 28028 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:47.858979 28028 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:47.859089 28028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:47.860180 28028 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:47.860600 28028 leader_election.cc:302] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4121fe0d80648a9bfe7ed669dc28f01; no voters:
I20260928 14:48:47.861002 28028 leader_election.cc:288] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:48:47.861258 28033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:47.861653 28033 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 LEADER]: Becoming Leader. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Running, Role: LEADER
I20260928 14:48:47.862053 28033 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:47.862764 28028 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:47.864193 28034 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:47.864374 28034 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:47.864881 28042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:47.865507 28035 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4121fe0d80648a9bfe7ed669dc28f01. Latest consensus state: current_term: 1 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:47.865633 28035 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:47.869279 28042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:47.876653 28042 catalog_manager.cc:1347] Generated new cluster ID: 20e4a720712a451bbdf1dafd64e51736
I20260928 14:48:47.876757 28042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:47.910991 28042 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:48:47.913079 28042 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:47.924252 28042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Generated new TSK 0
I20260928 14:48:47.925334 28042 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:48:47.976060 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:48.103991 28054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:48.104308 28054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:48.104355 28054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:48.108982 28054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:48.109196 28054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:48.109347 28054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:48:48.115811 28054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:48.117084 28054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:48.118299 28054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:48.129714 28060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:48.129755 28062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.129875 28054 server_base.cc:1034] running on GCE node
W20260928 14:48:48.129714 28059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.130638 28054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:48.131358 28054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:48.132575 28054 hybrid_clock.cc:648] HybridClock initialized: now 1790606928132549 us; error 35 us; skew 500 ppm
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:48.136629 28054 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:48:48.138304 28054 webserver.cc:460] Webserver started at http://127.13.223.65:33747/ using document root <none> and password file <none>
I20260928 14:48:48.139133 28054 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:48.139256 28054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:48.139537 28054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:48.142251 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "52fa3a640bee4f4e817fe247cb2754eb"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "f5280f674d5f1dd3ccc1feae894f496b"
server_key_iv: "5d708d6fc7c9e17caa91c96679668aa4"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.143303 28054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "52fa3a640bee4f4e817fe247cb2754eb"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "f5280f674d5f1dd3ccc1feae894f496b"
server_key_iv: "5d708d6fc7c9e17caa91c96679668aa4"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.148279 28054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260928 14:48:48.151350 28069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.152839 28054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:48:48.153014 28054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "52fa3a640bee4f4e817fe247cb2754eb"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "f5280f674d5f1dd3ccc1feae894f496b"
server_key_iv: "5d708d6fc7c9e17caa91c96679668aa4"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.153167 28054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:48.165400 28054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:48.169219 28054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:48.169391 28054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:48.169953 28054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:48.170898 28054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:48.170948 28054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.170991 28054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:48.171007 28054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.181623 28054 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:33323
I20260928 14:48:48.181651 28182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:33323 every 8 connection(s)
I20260928 14:48:48.182793 28054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:48.184801 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28054
I20260928 14:48:48.184939 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:48:48.185247 14205 external_mini_cluster.cc:1428] Setting key df02254d677537f9e6ebd484a3656341
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:48:48.201794 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.185066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:46965 (local address 127.13.223.126:35385)
0928 14:48:48.185306 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.185314 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.186353 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 14:48:48.188291 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.188293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.188347 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.188451 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:48.189998 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.191062 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.192743 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.193290 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.193449 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.196807 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:48.196814 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:48.196821 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:48.196866 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:48.198742 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.199535 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.199539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.199541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.199638 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.200088 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.200096 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.200099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.200350 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:48.200479 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.201296 (+ 817us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.201473 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:48:48.202732 28185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.185576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:46965)
0928 14:48:48.186171 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.186233 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.188030 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.188678 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.188691 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.189234 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:48.189803 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.189819 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.191263 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.191266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.192556 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.192570 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.193507 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.193510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.193727 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.194476 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:48.194495 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:48.196558 (+ 2063us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:48.198903 (+ 2345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.198911 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.198936 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.199419 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.199754 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.199758 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.199760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.199943 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.200482 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:48.200493 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:48.200956 (+ 463us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.201218 (+ 262us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.201593 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":158,"threads_started":1}
I20260928 14:48:48.204397 28183 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
I20260928 14:48:48.204766 28183 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:48.205468 28183 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:48.207261 27992 ts_manager.cc:194] Registered new tserver with Master: 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323)
I20260928 14:48:48.208572 27992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:46965
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:48:48.224603 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:48.354923 28188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:48.355264 28188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:48.355311 28188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:48.359992 28188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:48.360083 28188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:48.360148 28188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:48:48.366838 28188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:48.368578 28188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:48.370011 28188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:48.382977 28196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.383055 28188 server_base.cc:1034] running on GCE node
W20260928 14:48:48.383064 28194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:48.383296 28193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.383849 28188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:48.384644 28188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:48.385886 28188 hybrid_clock.cc:648] HybridClock initialized: now 1790606928385841 us; error 62 us; skew 500 ppm
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:48.389732 28188 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:48:48.390969 28188 webserver.cc:460] Webserver started at http://127.13.223.66:33961/ using document root <none> and password file <none>
I20260928 14:48:48.391745 28188 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:48.391837 28188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:48.392153 28188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:48.395153 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "86bb093d0142ee94b984fb786c14a2a0"
server_key_iv: "2613fbe5d0c3c595e4f8e993334acd1d"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.395902 28188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "86bb093d0142ee94b984fb786c14a2a0"
server_key_iv: "2613fbe5d0c3c595e4f8e993334acd1d"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.401010 28188 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260928 14:48:48.404433 28203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.406775 28188 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:48:48.406944 28188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "86bb093d0142ee94b984fb786c14a2a0"
server_key_iv: "2613fbe5d0c3c595e4f8e993334acd1d"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.407177 28188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:48.426954 28188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:48.430883 28188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:48.431103 28188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:48.431792 28188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:48.432730 28188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:48.432808 28188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.432878 28188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:48.432925 28188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.445364 28188 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37623
I20260928 14:48:48.445393 28316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37623 every 8 connection(s)
I20260928 14:48:48.446441 28188 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:48.453614 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28188
I20260928 14:48:48.453759 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:48:48.454077 14205 external_mini_cluster.cc:1428] Setting key ac9123172b68c4be93aed152463e888a
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:48:48.467278 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.449359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:42741 (local address 127.13.223.126:35385)
0928 14:48:48.449548 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.449552 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.450610 (+ 1058us) server_negotiation.cc:407] Connection header received
0928 14:48:48.452337 (+ 1727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.452341 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.452415 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.452548 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:48.454428 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.455154 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.456636 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.456982 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.457115 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.462055 (+ 4940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:48.462059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:48.462061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:48.462093 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:48.464471 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.465236 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.465240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.465243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.465347 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.465790 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.465794 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.465797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.466038 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:48.466174 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.466916 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.467082 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260928 14:48:48.468431 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.449777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:42741)
0928 14:48:48.450366 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.450423 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.452026 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.452781 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.452790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.453238 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:48.454202 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.454230 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.455315 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.455319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.456458 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.456470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.457181 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.457185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.457457 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.458911 (+ 1454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:48.458944 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:48.461777 (+ 2833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:48.464682 (+ 2905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.464691 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.464714 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.465078 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.465467 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.465470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.465472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.465666 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.466180 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:48.466191 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:48.466523 (+ 332us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.466838 (+ 315us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.467314 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":139,"threads_started":1}
I20260928 14:48:48.470455 28317 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:48:48.470904 28317 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:48.471664 28317 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:48.473436 27992 ts_manager.cc:194] Registered new tserver with Master: 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623)
I20260928 14:48:48.474143 27992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:42741
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:48:48.492158 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:48.626855 28322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:48.627230 28322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:48.627312 28322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:48.631539 28322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:48.631666 28322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:48.631770 28322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:48:48.637504 28322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:48.638803 28322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:48.639921 28322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:48.650606 28328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:48.650652 28330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.651058 28322 server_base.cc:1034] running on GCE node
W20260928 14:48:48.651096 28327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:48.651562 28322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:48.652314 28322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:48.653561 28322 hybrid_clock.cc:648] HybridClock initialized: now 1790606928653550 us; error 28 us; skew 500 ppm
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:48.657136 28322 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:48:48.658315 28322 webserver.cc:460] Webserver started at http://127.13.223.67:40731/ using document root <none> and password file <none>
I20260928 14:48:48.659003 28322 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:48.659071 28322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:48.659307 28322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:48:48.661648 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "758645f666aa4a4e9719545817addc46"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "b1017585b7c18a20e6a03268a3a7e3fb"
server_key_iv: "5cf606878c6ca7aca9d3208d437f87a2"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.662293 28322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "758645f666aa4a4e9719545817addc46"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "b1017585b7c18a20e6a03268a3a7e3fb"
server_key_iv: "5cf606878c6ca7aca9d3208d437f87a2"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.667181 28322 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 14:48:48.670267 28337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.672123 28322 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:48.672313 28322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "758645f666aa4a4e9719545817addc46"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "b1017585b7c18a20e6a03268a3a7e3fb"
server_key_iv: "5cf606878c6ca7aca9d3208d437f87a2"
server_key_version: "encryptionkey@0"
I20260928 14:48:48.672465 28322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:48.697234 28322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:48.701336 28322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:48.701658 28322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:48.702453 28322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:48.703622 28322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:48.703686 28322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.703779 28322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:48.703824 28322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:48.716019 28322 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40455
I20260928 14:48:48.716042 28450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40455 every 8 connection(s)
I20260928 14:48:48.717182 28322 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:48.721447 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28322
I20260928 14:48:48.721557 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:48:48.721887 14205 external_mini_cluster.cc:1428] Setting key 9b2b5faf9deba00acc8a1842898dc9d1
Sep 28 14:48:48 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606928, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:48.733942 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.719599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:50265 (local address 127.13.223.126:35385)
0928 14:48:48.719762 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.719767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.720775 (+ 1008us) server_negotiation.cc:407] Connection header received
0928 14:48:48.722325 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.722329 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.722390 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.722499 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:48.723975 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.724747 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.725966 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.726241 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.726331 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.729632 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:48.729636 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:48.729639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:48.729669 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:48.731438 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.732069 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.732073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.732074 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.732173 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.732675 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.732678 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.732679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.732897 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:48.732987 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.733674 (+ 687us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.733803 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:48:48.735144 28453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.720055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:50265)
0928 14:48:48.720569 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.720646 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.722086 (+ 1440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.722758 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.722768 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.723207 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:48.723821 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.723842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.724874 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.724877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.725842 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.725852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.726522 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.726525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.726804 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.727487 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:48.727505 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:48.729360 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:48.731587 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.731594 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.731640 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.731937 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.732309 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:48.732312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:48.732314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:48.732557 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:48.733033 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:48.733042 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:48.733360 (+ 318us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.733612 (+ 252us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.733935 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260928 14:48:48.736554 28451 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
I20260928 14:48:48.736837 28451 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:48.737438 28451 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:48.738701 27992 ts_manager.cc:194] Registered new tserver with Master: 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
I20260928 14:48:48.739363 27992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:50265
I20260928 14:48:48.748973 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:48.761200 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.751219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57132 (local address 127.13.223.126:35385)
0928 14:48:48.751386 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.751394 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.751783 (+ 389us) server_negotiation.cc:407] Connection header received
0928 14:48:48.751906 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.751909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.751961 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.752044 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:48.752906 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.753541 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.754498 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.754891 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.754997 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.756829 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:48.756833 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:48.756835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:48.756896 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:48.758816 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.759438 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.759442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.759443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.759530 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:48.759914 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:48.759917 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:48.759919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:48.760283 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:48.760405 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.760775 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.760888 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:48:48.765599 27992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:48.768960 27992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:48.786293 28462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.779649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.0.0.1:56462)
0928 14:48:48.780260 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.780317 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.780492 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.781157 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.781164 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.781188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.781372 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.781379 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.782867 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.782869 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.784185 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.784195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.785545 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.785549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.785705 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.785754 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.785939 (+ 185us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.786112 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":154,"threads_started":1}
I20260928 14:48:48.786298 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.779813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56462 (local address 127.13.223.66:37623)
0928 14:48:48.780258 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.780271 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.780393 (+ 122us) server_negotiation.cc:407] Connection header received
0928 14:48:48.780698 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.780704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.780964 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.781157 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.781505 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.782720 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.784331 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.785316 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.785522 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.785705 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.785977 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.786109 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":171,"threads_started":1}
I20260928 14:48:48.787310 28464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.780572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.0.0.1:39106)
0928 14:48:48.780897 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.780915 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.781005 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.782330 (+ 1325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.782336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.782354 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.782461 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.782466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.784023 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.784026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.785285 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.785296 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.786802 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.786806 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.786939 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.786973 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.787064 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.787164 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":134,"threads_started":1}
I20260928 14:48:48.787617 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.780775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.13.223.65:33323)
0928 14:48:48.781660 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.781667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.781692 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:48:48.781772 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.781777 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.781982 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.782161 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.782643 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.783900 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.785443 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.786410 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.786535 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.786733 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.787136 (+ 403us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.787254 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":109,"threads_started":1}
I20260928 14:48:48.788434 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.780981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40455 (local address 127.0.0.1:53378)
0928 14:48:48.781626 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.781640 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.781728 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.783582 (+ 1854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.783585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.783606 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.783747 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.783754 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.785284 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.785288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.786670 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.786683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.787993 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.787998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.788096 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.788121 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.788212 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.788289 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":95,"threads_started":1}
I20260928 14:48:48.788695 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.781391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.13.223.67:40455)
0928 14:48:48.783058 (+ 1667us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.783070 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.783095 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:48:48.783175 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.783180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.783379 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.783564 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.783885 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.785108 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.786830 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.787863 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.787993 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.788142 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.788350 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.788476 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1551,"thread_start_us":97,"threads_started":1}
I20260928 14:48:48.790675 28251 tablet_service.cc:1467] Processing CreateTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 (DEFAULT_TABLE table=test-table [id=dbe371684b1742058cd80e24e99ad34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:48.790874 28117 tablet_service.cc:1467] Processing CreateTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 (DEFAULT_TABLE table=test-table [id=dbe371684b1742058cd80e24e99ad34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:48.791250 28385 tablet_service.cc:1467] Processing CreateTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 (DEFAULT_TABLE table=test-table [id=dbe371684b1742058cd80e24e99ad34a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:48.792021 28117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b986c639bcd41d6b917d35b82b881f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:48.792021 28251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b986c639bcd41d6b917d35b82b881f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:48.792323 28385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b986c639bcd41d6b917d35b82b881f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:48.799921 28468 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Bootstrap starting.
I20260928 14:48:48.801946 28469 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap starting.
I20260928 14:48:48.803031 28470 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Bootstrap starting.
I20260928 14:48:48.803511 28468 tablet_bootstrap.cc:654] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:48.804464 28469 tablet_bootstrap.cc:654] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:48.804812 28468 log.cc:824] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:48.805387 28469 log.cc:824] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:48.805992 28470 tablet_bootstrap.cc:654] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:48.807071 28470 log.cc:824] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:48.807606 28468 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: No bootstrap required, opened a new log
I20260928 14:48:48.807900 28468 ts_tablet_manager.cc:1397] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260928 14:48:48.808283 28469 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: No bootstrap required, opened a new log
I20260928 14:48:48.808524 28469 ts_tablet_manager.cc:1397] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:48:48.809828 28470 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: No bootstrap required, opened a new log
I20260928 14:48:48.810034 28470 ts_tablet_manager.cc:1397] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:48:48.811949 28468 raft_consensus.cc:348] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.812345 28468 raft_consensus.cc:374] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:48.812412 28468 raft_consensus.cc:729] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758645f666aa4a4e9719545817addc46, State: Initialized, Role: FOLLOWER
I20260928 14:48:48.812947 28468 consensus_queue.cc:260] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.813774 28451 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:48.814286 28468 ts_tablet_manager.cc:1428] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:48:48.814401 28469 raft_consensus.cc:348] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.815081 28469 raft_consensus.cc:374] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:48.815160 28469 raft_consensus.cc:729] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fa3a640bee4f4e817fe247cb2754eb, State: Initialized, Role: FOLLOWER
I20260928 14:48:48.815678 28469 consensus_queue.cc:260] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.816054 28470 raft_consensus.cc:348] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.816425 28183 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:48.816443 28470 raft_consensus.cc:374] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:48.816605 28470 raft_consensus.cc:729] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cab114e7ed4ff6b74565c0c69bcc5c, State: Initialized, Role: FOLLOWER
I20260928 14:48:48.816998 28469 ts_tablet_manager.cc:1428] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent starting tablet: real 0.008s user 0.003s sys 0.004s
I20260928 14:48:48.817309 28470 consensus_queue.cc:260] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.818094 28317 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:48.818594 28470 ts_tablet_manager.cc:1428] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
I20260928 14:48:48.825556 28476 raft_consensus.cc:482] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:48.825794 28476 raft_consensus.cc:504] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.827291 28476 leader_election.cc:288] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
I20260928 14:48:48.832231 28319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.827562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.13.223.66:38831)
0928 14:48:48.827725 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.827744 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.827851 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.828334 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.828338 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.828370 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.828583 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.828597 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.829719 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.829729 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.830697 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.830707 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.831623 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.831628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.831765 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.831794 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.831899 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.831969 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":39}
I20260928 14:48:48.832226 28465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.827891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:38831 (local address 127.13.223.65:33323)
0928 14:48:48.828087 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.828092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.828108 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:48:48.828168 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.828175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.828231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.828302 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.828740 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.829587 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.830859 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.831479 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.831596 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.831653 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.831951 (+ 298us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.832077 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260928 14:48:48.832365 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.828066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:36939 (local address 127.13.223.67:40455)
0928 14:48:48.828222 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.828228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.828432 (+ 204us) server_negotiation.cc:407] Connection header received
0928 14:48:48.828611 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.828613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.828728 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.828820 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.829212 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.829808 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.831089 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.831649 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.831781 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.831835 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.832149 (+ 314us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.832233 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:48:48.832413 28477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.827897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40455 (local address 127.13.223.66:36939)
0928 14:48:48.828320 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.828336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.828462 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.828870 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.828872 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.828890 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.828991 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.828996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.829950 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.829953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.830859 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.830870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.831808 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.831811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.831983 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.832009 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.832115 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.832190 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":145,"threads_started":1}
I20260928 14:48:48.832968 28475 raft_consensus.cc:482] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:48.833211 28475 raft_consensus.cc:504] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.833568 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa3a640bee4f4e817fe247cb2754eb" is_pre_election: true
I20260928 14:48:48.833590 28405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758645f666aa4a4e9719545817addc46" is_pre_election: true
I20260928 14:48:48.833935 28405 raft_consensus.cc:2454] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in term 0.
I20260928 14:48:48.834458 28206 leader_election.cc:302] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c; no voters:
I20260928 14:48:48.834583 28137 raft_consensus.cc:2454] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in term 0.
I20260928 14:48:48.834846 28476 raft_consensus.cc:2781] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:48.834931 28476 raft_consensus.cc:482] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:48.834975 28476 raft_consensus.cc:3037] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:48.835762 28475 leader_election.cc:288] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
I20260928 14:48:48.836591 28476 raft_consensus.cc:504] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.837294 28137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa3a640bee4f4e817fe247cb2754eb"
I20260928 14:48:48.837448 28137 raft_consensus.cc:3037] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:48.839008 28137 raft_consensus.cc:2454] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in term 1.
I20260928 14:48:48.839432 28206 leader_election.cc:302] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb, 76cab114e7ed4ff6b74565c0c69bcc5c; no voters:
I20260928 14:48:48.840270 28405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758645f666aa4a4e9719545817addc46"
I20260928 14:48:48.840415 28405 raft_consensus.cc:3037] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:48.841105 28478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.835093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.13.223.65:38311)
0928 14:48:48.836008 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.836024 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.836189 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.836635 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.836639 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.836657 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.836791 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.836796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.837862 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.837866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.839454 (+ 1588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.839466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.840576 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.840579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.840692 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.840749 (+ 57us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.840826 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.840909 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"mutex_wait_us":60,"thread_start_us":137,"threads_started":1}
I20260928 14:48:48.841591 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.835502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:38311 (local address 127.13.223.66:37623)
0928 14:48:48.835653 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.835658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.836103 (+ 445us) server_negotiation.cc:407] Connection header received
0928 14:48:48.836326 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.836330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.836382 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.836435 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.836914 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.837734 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.839614 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.840302 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.840413 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.840474 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.841358 (+ 884us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.841457 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:48:48.842056 28467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.836406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:49413 (local address 127.13.223.67:40455)
0928 14:48:48.836582 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.836588 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.837177 (+ 589us) server_negotiation.cc:407] Connection header received
0928 14:48:48.837363 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.837366 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.837412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.837459 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:48.837864 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.838748 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.840443 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.841152 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.841263 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.841336 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.841836 (+ 500us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.841928 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:48:48.842038 28405 raft_consensus.cc:2454] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in term 1.
I20260928 14:48:48.842396 28271 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "52fa3a640bee4f4e817fe247cb2754eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" is_pre_election: true
I20260928 14:48:48.842715 28271 raft_consensus.cc:2379] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52fa3a640bee4f4e817fe247cb2754eb in current term 1: Already voted for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in this term.
I20260928 14:48:48.842733 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.835981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40455 (local address 127.13.223.65:49413)
0928 14:48:48.837086 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.837102 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:48.837245 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:48.837586 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:48.837589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:48.837617 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:48.837727 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.837733 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.838915 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.838918 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.840278 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:48.840291 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.841441 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.841445 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:48.841616 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.841646 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:48:48.842387 (+ 741us) client_negotiation.cc:240] Negotiation successful
0928 14:48:48.842485 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1036,"mutex_wait_us":1,"thread_start_us":126,"threads_started":1}
I20260928 14:48:48.843178 28405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b986c639bcd41d6b917d35b82b881f6" candidate_uuid: "52fa3a640bee4f4e817fe247cb2754eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "758645f666aa4a4e9719545817addc46" is_pre_election: true
I20260928 14:48:48.843381 28405 raft_consensus.cc:2379] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 52fa3a640bee4f4e817fe247cb2754eb in current term 1: Already voted for candidate 76cab114e7ed4ff6b74565c0c69bcc5c in this term.
I20260928 14:48:48.843472 28476 leader_election.cc:288] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [CANDIDATE]: Term 1 election: Requested vote from peers 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
I20260928 14:48:48.843681 28476 raft_consensus.cc:2781] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:48.843771 28476 raft_consensus.cc:686] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 LEADER]: Becoming Leader. State: Replica: 76cab114e7ed4ff6b74565c0c69bcc5c, State: Running, Role: LEADER
I20260928 14:48:48.843942 28072 leader_election.cc:302] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:48:48.844115 28476 consensus_queue.cc:237] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:48.844254 28475 raft_consensus.cc:2726] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260928 14:48:48.847858 27991 catalog_manager.cc:5495] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c reported cstate change: term changed from 0 to 1, leader changed from <none> to 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66). New cstate: current_term: 1 leader_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } health_report { overall_health: UNKNOWN } } }
I20260928 14:48:48.875667 28463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.871631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56474 (local address 127.13.223.66:37623)
0928 14:48:48.871774 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:48.871778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:48.871792 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:48:48.871929 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:48.871931 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:48.871986 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:48.872085 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:48.872412 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.872950 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.873886 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:48.874126 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:48.874213 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:48.874696 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:48.875176 (+ 480us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:48.875279 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:48.875405 (+ 126us) server_negotiation.cc:299] Negotiation successful
0928 14:48:48.875460 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:48:48.882629 28137 raft_consensus.cc:1261] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Refusing update from remote peer 76cab114e7ed4ff6b74565c0c69bcc5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:48.882627 28405 raft_consensus.cc:1261] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Refusing update from remote peer 76cab114e7ed4ff6b74565c0c69bcc5c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:48.883376 28476 consensus_queue.cc:1035] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:48.883639 28480 consensus_queue.cc:1035] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:48.903643 27992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:48.903910 27992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:48.907351 27992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Sending DeleteTablet for 3 replicas of tablet 2b986c639bcd41d6b917d35b82b881f6
I20260928 14:48:48.908437 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28054
I20260928 14:48:48.908362 28251 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:56462
I20260928 14:48:48.909155 28385 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:53378
I20260928 14:48:48.909366 28493 tablet_replica.cc:330] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: stopping tablet replica
I20260928 14:48:48.909968 28493 raft_consensus.cc:2229] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:48.910600 28493 raft_consensus.cc:2258] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:48.911087 28494 tablet_replica.cc:330] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: stopping tablet replica
I20260928 14:48:48.911679 28494 raft_consensus.cc:2229] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:48.912072 28494 raft_consensus.cc:2258] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:48.913345 28493 ts_tablet_manager.cc:1905] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:48.913968 28494 ts_tablet_manager.cc:1905] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:48.917373 28494 ts_tablet_manager.cc:1918] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:48.917513 28494 log.cc:1196] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/2b986c639bcd41d6b917d35b82b881f6
I20260928 14:48:48.917706 28493 ts_tablet_manager.cc:1918] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:48.917915 28493 log.cc:1196] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/2b986c639bcd41d6b917d35b82b881f6
I20260928 14:48:48.917925 28494 ts_tablet_manager.cc:1939] T 2b986c639bcd41d6b917d35b82b881f6 P 758645f666aa4a4e9719545817addc46: Deleting consensus metadata
I20260928 14:48:48.918372 28493 ts_tablet_manager.cc:1939] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting consensus metadata
W20260928 14:48:48.918617 27978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:33323 (error 108)
I20260928 14:48:48.918712 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28188
I20260928 14:48:48.920382 27978 catalog_manager.cc:4841] TS 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): tablet 2b986c639bcd41d6b917d35b82b881f6 (table test-table [id=dbe371684b1742058cd80e24e99ad34a]) successfully deleted
I20260928 14:48:48.921226 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.920801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.0.0.1:39110)
0928 14:48:48.921027 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.921115 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
W20260928 14:48:48.921466 27978 catalog_manager.cc:4584] TS 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2b986c639bcd41d6b917d35b82b881f6: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33323: connect: Connection refused (error 111)
I20260928 14:48:48.926170 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28322
I20260928 14:48:48.928184 28466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:48.927789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.0.0.1:56484)
0928 14:48:48.928017 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:48.928085 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 14:48:48.932466 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 27960
I20260928 14:48:48.940052 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--webserver_interface=127.13.223.126
--webserver_port=43549
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:35385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:49.065424 28496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:49.065757 28496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:49.065798 28496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:49.070109 28496 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:49.070197 28496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:49.070218 28496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:49.070236 28496 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:49.070252 28496 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:49.076078 28496 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:35385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=43549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:49.077241 28496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:49.078240 28496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:49.094973 28503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.094928 28505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:49.095117 28496 server_base.cc:1034] running on GCE node
W20260928 14:48:49.094987 28502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:49.096061 28496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:49.097478 28496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:49.098644 28496 hybrid_clock.cc:648] HybridClock initialized: now 1790606929098608 us; error 65 us; skew 500 ppm
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:49.102015 28496 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:49.103313 28496 webserver.cc:460] Webserver started at http://127.13.223.126:43549/ using document root <none> and password file <none>
I20260928 14:48:49.104012 28496 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:49.104146 28496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:49.109221 28496 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:48:49.112210 28512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.113994 28496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:48:49.114205 28496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c4121fe0d80648a9bfe7ed669dc28f01"
format_stamp: "Formatted at 2026-09-28 14:48:47 on dist-test-slave-gkhx"
server_key: "2e8aec4be54d39da0ce725ebdd7a85f0"
server_key_iv: "ee5ffb0c23fbe5de459ec82c507df686"
server_key_version: "encryptionkey@0"
I20260928 14:48:49.115029 28496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:49.130048 28496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:49.133912 28496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:49.134266 28496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:49.142447 28496 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:35385
I20260928 14:48:49.142838 28564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:35385 every 8 connection(s)
I20260928 14:48:49.143730 28496 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:49.148693 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28496
I20260928 14:48:49.149072 28565 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:49.149325 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:33323
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=33747
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:49.152009 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap starting.
I20260928 14:48:49.172788 28565 log.cc:824] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:49.182909 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:49.183537 28565 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap complete.
I20260928 14:48:49.188012 28565 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:49.188467 28565 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Initialized, Role: FOLLOWER
I20260928 14:48:49.189060 28565 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:49.189186 28565 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:49.189268 28565 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:49.189392 28565 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:48:49.191577 28565 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:49.191956 28565 leader_election.cc:302] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4121fe0d80648a9bfe7ed669dc28f01; no voters:
I20260928 14:48:49.192324 28565 leader_election.cc:288] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:48:49.192441 28569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:48:49.192669 28569 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 LEADER]: Becoming Leader. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Running, Role: LEADER
I20260928 14:48:49.193049 28569 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:49.193717 28565 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:49.194591 28571 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4121fe0d80648a9bfe7ed669dc28f01. Latest consensus state: current_term: 2 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:49.194731 28571 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:49.194671 28570 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:49.194790 28570 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:49.195106 28577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:49.200323 28577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:49.201191 28577 catalog_manager.cc:1259] Loaded cluster ID: 20e4a720712a451bbdf1dafd64e51736
I20260928 14:48:49.201252 28577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:48:49.202795 28577 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 14:48:49.207891 28577 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 14:48:49.214114 28577 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:48:49.215188 28577 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Loaded TSK: 0
I20260928 14:48:49.216509 28577 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:48:49.283259 28567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:49.283706 28567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:49.283802 28567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:49.292716 28567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:49.292826 28567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:49.292965 28567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:48:49.305176 28567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:33323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=33747
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:49.306710 28567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:49.308100 28567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:49.322321 28592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.322321 28593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.322855 28595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:49.323766 28567 server_base.cc:1034] running on GCE node
I20260928 14:48:49.324405 28567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:49.325254 28567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:49.326457 28567 hybrid_clock.cc:648] HybridClock initialized: now 1790606929326413 us; error 65 us; skew 500 ppm
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:49.338181 28567 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:48:49.339563 28567 webserver.cc:460] Webserver started at http://127.13.223.65:33747/ using document root <none> and password file <none>
I20260928 14:48:49.340240 28567 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:49.340366 28567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:49.345059 28567 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260928 14:48:49.347678 28602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.349304 28567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:48:49.349493 28567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "52fa3a640bee4f4e817fe247cb2754eb"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "f5280f674d5f1dd3ccc1feae894f496b"
server_key_iv: "5d708d6fc7c9e17caa91c96679668aa4"
server_key_version: "encryptionkey@0"
I20260928 14:48:49.350180 28567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:49.374159 28567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:49.378357 28567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:49.378646 28567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:49.379387 28567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:49.380854 28609 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:48:49.385515 28567 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:48:49.385637 28567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 14:48:49.385720 28567 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:48:49.387748 28567 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:48:49.387823 28567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 14:48:49.388242 28609 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap starting.
I20260928 14:48:49.400072 28567 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:33323
I20260928 14:48:49.400386 28716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:33323 every 8 connection(s)
I20260928 14:48:49.401728 28567 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:49.408481 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28567
I20260928 14:48:49.409080 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:37623
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=33961
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:49.412858 28609 log.cc:824] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:49.422222 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.404466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:44973 (local address 127.13.223.126:35385)
0928 14:48:49.404872 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.404896 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.405434 (+ 538us) server_negotiation.cc:407] Connection header received
0928 14:48:49.407138 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.407149 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.407479 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.407841 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:49.409202 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.410311 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.412272 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.412793 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.413006 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.416195 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:49.416202 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:49.416216 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:49.416247 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:49.418369 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.419218 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.419224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.419228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.419362 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.419813 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.419816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.419818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.420283 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:49.420440 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.421048 (+ 608us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.421341 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":145,"threads_started":1}
I20260928 14:48:49.422240 28720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.404553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:44973)
0928 14:48:49.405231 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.405299 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.406670 (+ 1371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.407906 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.407914 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.408378 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:49.408971 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.408989 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.410512 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.410515 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.412116 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.412134 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.412963 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.412966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.413234 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.413999 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:49.414019 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:49.415957 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:49.418616 (+ 2659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.418625 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.418640 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.419071 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.419496 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.419499 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.419502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.419682 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.420411 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:49.420418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:49.420687 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.420908 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.421279 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":178,"threads_started":1}
I20260928 14:48:49.424959 28717 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
I20260928 14:48:49.425231 28717 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:49.425796 28717 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:49.425855 28609 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:49.426425 28609 tablet_bootstrap.cc:492] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap complete.
I20260928 14:48:49.428081 28609 ts_tablet_manager.cc:1397] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent bootstrapping tablet: real 0.040s user 0.028s sys 0.009s
I20260928 14:48:49.428542 28529 ts_manager.cc:194] Registered new tserver with Master: 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323)
I20260928 14:48:49.431527 28609 raft_consensus.cc:348] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:49.432286 28609 raft_consensus.cc:729] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fa3a640bee4f4e817fe247cb2754eb, State: Initialized, Role: FOLLOWER
I20260928 14:48:49.432348 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:44973
I20260928 14:48:49.433303 28609 consensus_queue.cc:260] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } }
I20260928 14:48:49.434465 28609 ts_tablet_manager.cc:1428] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:48:49.435603 28717 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:49.440141 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.431667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.0.0.1:39114)
0928 14:48:49.433773 (+ 2106us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.433825 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.433962 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.435091 (+ 1129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.435097 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.435123 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:49.435254 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.435262 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.436908 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.436912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.438332 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.438355 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.439582 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.439585 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.439678 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.439712 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.439826 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.439934 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742,"thread_start_us":144,"threads_started":1}
I20260928 14:48:49.440171 28724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.432563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39114 (local address 127.13.223.65:33323)
0928 14:48:49.434251 (+ 1688us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.434354 (+ 103us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.434382 (+ 28us) server_negotiation.cc:407] Connection header received
0928 14:48:49.434457 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.434462 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.434765 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.434936 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:49.435423 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.436744 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.438498 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.439418 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.439552 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.439713 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.439900 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.440018 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1587,"thread_start_us":116,"threads_started":1}
I20260928 14:48:49.441459 28651 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b986c639bcd41d6b917d35b82b881f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:48 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39114
I20260928 14:48:49.442618 28726 tablet_replica.cc:330] stopping tablet replica
I20260928 14:48:49.442973 28726 raft_consensus.cc:2229] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:49.443254 28726 raft_consensus.cc:2258] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:49.444512 28726 ts_tablet_manager.cc:1905] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:49.449913 28726 ts_tablet_manager.cc:1918] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:49.450059 28726 log.cc:1196] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/2b986c639bcd41d6b917d35b82b881f6
I20260928 14:48:49.450347 28726 ts_tablet_manager.cc:1939] T 2b986c639bcd41d6b917d35b82b881f6 P 52fa3a640bee4f4e817fe247cb2754eb: Deleting consensus metadata
I20260928 14:48:49.452704 28515 catalog_manager.cc:4841] TS 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323): tablet 2b986c639bcd41d6b917d35b82b881f6 (table test-table [id=dbe371684b1742058cd80e24e99ad34a]) successfully deleted
W20260928 14:48:49.557555 28721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:49.557976 28721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:49.558045 28721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:49.564791 28721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:49.564895 28721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:49.565014 28721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:48:49.571982 28721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=33961
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:49.573282 28721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:49.574409 28721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:48:49.585532 28721 server_base.cc:1034] running on GCE node
W20260928 14:48:49.585454 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.585419 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.585469 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:49.586177 28721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:49.586963 28721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:49.588189 28721 hybrid_clock.cc:648] HybridClock initialized: now 1790606929588159 us; error 48 us; skew 500 ppm
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:49.591548 28721 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:48:49.592659 28721 webserver.cc:460] Webserver started at http://127.13.223.66:33961/ using document root <none> and password file <none>
I20260928 14:48:49.593235 28721 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:49.593313 28721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:49.597973 28721 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:49.600641 28741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.602291 28721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:49.602425 28721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "86bb093d0142ee94b984fb786c14a2a0"
server_key_iv: "2613fbe5d0c3c595e4f8e993334acd1d"
server_key_version: "encryptionkey@0"
I20260928 14:48:49.603202 28721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:49.638091 28721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:49.641877 28721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:49.642134 28721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:49.642964 28721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:49.644258 28748 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:48:49.648978 28748 ts_tablet_manager.cc:1852] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:48:49.649509 28748 ts_tablet_manager.cc:1905] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:49.653908 28748 ts_tablet_manager.cc:1918] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:49.654134 28748 ts_tablet_manager.cc:1939] T 2b986c639bcd41d6b917d35b82b881f6 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting consensus metadata
I20260928 14:48:49.654748 28721 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:48:49.654832 28721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.001s sys 0.000s
I20260928 14:48:49.654947 28721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:49.654991 28721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.665678 28721 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37623
I20260928 14:48:49.665722 28855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37623 every 8 connection(s)
I20260928 14:48:49.666742 28721 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:49.673192 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28721
I20260928 14:48:49.673664 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:40455
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=40731
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:49.683158 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.668917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:58079 (local address 127.13.223.126:35385)
0928 14:48:49.669089 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.669093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.670035 (+ 942us) server_negotiation.cc:407] Connection header received
0928 14:48:49.671284 (+ 1249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.671287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.671350 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.671442 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:49.673108 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.673762 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.675417 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.675680 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.675818 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.679096 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:49.679099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:49.679102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:49.679130 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:49.680739 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.681377 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.681386 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.681387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.681466 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.681847 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.681851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.681852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.682069 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:49.682184 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.682802 (+ 618us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.682959 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:48:49.684028 28858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.669297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:58079)
0928 14:48:49.669849 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.669900 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.670901 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.671672 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.671679 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.672143 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:49.672851 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.672877 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.673968 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.673972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.675278 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.675289 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.675869 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.675874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.676067 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.676783 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:49.676803 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:49.678863 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:49.680910 (+ 2047us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.680918 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.680938 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.681250 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.681571 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.681574 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.681576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.681738 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.682184 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:49.682192 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:49.682474 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.682742 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.683112 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":127,"threads_started":1}
I20260928 14:48:49.685390 28856 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
I20260928 14:48:49.685714 28856 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:49.686264 28856 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:49.687493 28529 ts_manager.cc:194] Registered new tserver with Master: 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623)
I20260928 14:48:49.688374 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:58079
W20260928 14:48:49.799952 28859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:49.800273 28859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:49.800319 28859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:49.805087 28859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:49.805190 28859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:49.805255 28859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:48:49.811113 28859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:40455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=40731
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:49.812529 28859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:49.813726 28859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:49.825446 28865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.825446 28867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:49.825510 28864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:49.825908 28859 server_base.cc:1034] running on GCE node
I20260928 14:48:49.826692 28859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:49.827430 28859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:49.828655 28859 hybrid_clock.cc:648] HybridClock initialized: now 1790606929828619 us; error 51 us; skew 500 ppm
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:49.831943 28859 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:48:49.833213 28859 webserver.cc:460] Webserver started at http://127.13.223.67:40731/ using document root <none> and password file <none>
I20260928 14:48:49.833979 28859 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:49.834098 28859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:49.838413 28859 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:49.840912 28874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.842468 28859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:49.842676 28859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "758645f666aa4a4e9719545817addc46"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "b1017585b7c18a20e6a03268a3a7e3fb"
server_key_iv: "5cf606878c6ca7aca9d3208d437f87a2"
server_key_version: "encryptionkey@0"
I20260928 14:48:49.843279 28859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:49.859215 28859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:49.863137 28859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:49.863373 28859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:49.864023 28859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:49.864910 28859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:49.864985 28859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.865056 28859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:49.865104 28859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:49.875669 28859 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40455
I20260928 14:48:49.875691 28987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40455 every 8 connection(s)
I20260928 14:48:49.876796 28859 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:49.882401 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 28859
Sep 28 14:48:49 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606929, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:48:49.892055 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.883282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57136 (local address 127.13.223.126:35385)
0928 14:48:49.883589 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.883593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.883732 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:48:49.883902 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.883905 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.883952 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.884056 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:49.884800 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.885326 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.886202 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.886479 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.886638 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.888181 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:49.888184 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:49.888186 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:49.888210 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:49.890068 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.890590 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.890595 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.890600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.890700 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.891036 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.891039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.891041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.891256 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:49.891493 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.891709 (+ 216us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.891836 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":130,"threads_started":1}
I20260928 14:48:49.893409 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.879204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:60825 (local address 127.13.223.126:35385)
0928 14:48:49.879359 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.879363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.880416 (+ 1053us) server_negotiation.cc:407] Connection header received
0928 14:48:49.882034 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.882038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.882105 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.882228 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:49.884161 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.884696 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.885948 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.886183 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.886304 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.889157 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:49.889160 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:49.889162 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:49.889198 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:49.890765 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.891353 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.891358 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.891360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.891467 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:49.891911 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:49.891914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:49.891916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:49.892152 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:49.892268 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.893102 (+ 834us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.893238 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 14:48:49.894459 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.879582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:60825)
0928 14:48:49.880214 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.880268 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.881764 (+ 1496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.882442 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.882451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.883084 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:49.883918 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.883944 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.884837 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.884841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.885812 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.885821 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.886361 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.886365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.886607 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.887263 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:49.887281 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:49.888962 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:49.890910 (+ 1948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.890917 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.890936 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.891237 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.891586 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:49.891589 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:49.891591 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:49.891779 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:49.892325 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:49.892333 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:49.892707 (+ 374us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.892945 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.893268 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":108,"threads_started":1}
I20260928 14:48:49.895725 28988 heartbeater.cc:347] Connected to a master server at 127.13.223.126:35385
I20260928 14:48:49.896004 28988 heartbeater.cc:464] Registering TS with master...
I20260928 14:48:49.896548 28988 heartbeater.cc:511] Master 127.13.223.126:35385 requested a full tablet report, sending...
I20260928 14:48:49.897732 28529 ts_manager.cc:194] Registered new tserver with Master: 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
I20260928 14:48:49.898375 28529 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:60825
I20260928 14:48:49.907310 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:48:49.913131 28719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.908270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57148 (local address 127.13.223.126:35385)
0928 14:48:49.908490 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.908496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.908682 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:48:49.908909 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.908911 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.909038 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.909237 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:49.909513 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.910047 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.911207 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.911512 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.911624 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.912071 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:49.912588 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:49.912710 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.912872 (+ 162us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.912931 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 14:48:49.913933 28529 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:57148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:48:49.917590 28529 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:48:49.927604 28651 tablet_service.cc:1467] Processing CreateTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 (DEFAULT_TABLE table=test-table [id=f7b445e9a3d8408797fc293b917180ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:49.928350 28651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761af7ac93c94eb9b80e421c71fb8878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:49.931354 28723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.926045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.0.0.1:56486)
0928 14:48:49.926350 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.926370 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.926468 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.927333 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.927336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.927356 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:49.927475 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.927485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.929013 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.929016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.929916 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.929926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.930910 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.930914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.931007 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.931030 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.931102 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.931199 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 14:48:49.931399 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.926262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.13.223.66:37623)
0928 14:48:49.926767 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.926778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.926799 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:48:49.926875 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.926880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.927139 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.927364 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:49.927614 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.928848 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.930054 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.930754 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.930891 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.931064 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.931144 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.931237 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":122,"threads_started":1}
I20260928 14:48:49.932204 28609 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap starting.
I20260928 14:48:49.932945 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.926238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40455 (local address 127.0.0.1:53394)
0928 14:48:49.926505 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:49.926520 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:49.926642 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:49.928097 (+ 1455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:49.928100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:49.928116 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:49.928189 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.928194 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.929435 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.929438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.930743 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:49.930757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.932444 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.932448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:49.932570 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.932605 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:48:49.932693 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:48:49.932786 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":136,"threads_started":1}
I20260928 14:48:49.933149 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:49.926899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53394 (local address 127.13.223.67:40455)
0928 14:48:49.927280 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:49.927291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:49.927313 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:48:49.927574 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:49.927583 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:49.927770 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:49.927941 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:49.928305 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.929262 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.930909 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:49.932092 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:49.932208 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:49.932377 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:49.932838 (+ 461us) server_negotiation.cc:299] Negotiation successful
0928 14:48:49.932982 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":206,"threads_started":1}
I20260928 14:48:49.933568 28609 tablet_bootstrap.cc:654] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:49.934933 28609 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: No bootstrap required, opened a new log
I20260928 14:48:49.935034 28609 ts_tablet_manager.cc:1397] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:49.935026 28790 tablet_service.cc:1467] Processing CreateTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 (DEFAULT_TABLE table=test-table [id=f7b445e9a3d8408797fc293b917180ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:49.935519 28609 raft_consensus.cc:348] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.935652 28609 raft_consensus.cc:374] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:49.935695 28609 raft_consensus.cc:729] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fa3a640bee4f4e817fe247cb2754eb, State: Initialized, Role: FOLLOWER
I20260928 14:48:49.935842 28609 consensus_queue.cc:260] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.936245 28609 ts_tablet_manager.cc:1428] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 14:48:49.936383 28790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761af7ac93c94eb9b80e421c71fb8878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:49.941011 28922 tablet_service.cc:1467] Processing CreateTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 (DEFAULT_TABLE table=test-table [id=f7b445e9a3d8408797fc293b917180ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:48:49.942335 28922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 761af7ac93c94eb9b80e421c71fb8878. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:48:49.945194 28748 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Bootstrap starting.
I20260928 14:48:49.948285 28748 tablet_bootstrap.cc:654] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:49.949163 28748 log.cc:824] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:49.951613 28748 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: No bootstrap required, opened a new log
I20260928 14:48:49.951841 28748 ts_tablet_manager.cc:1397] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20260928 14:48:49.952512 29003 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Bootstrap starting.
I20260928 14:48:49.955760 29003 tablet_bootstrap.cc:654] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Neither blocks nor log segments found. Creating new log.
I20260928 14:48:49.956436 28748 raft_consensus.cc:348] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.956777 28748 raft_consensus.cc:374] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:49.956754 29003 log.cc:824] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:49.956828 28748 raft_consensus.cc:729] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76cab114e7ed4ff6b74565c0c69bcc5c, State: Initialized, Role: FOLLOWER
I20260928 14:48:49.957409 28748 consensus_queue.cc:260] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.958640 28748 ts_tablet_manager.cc:1428] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 14:48:49.959050 28856 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:49.959398 29003 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: No bootstrap required, opened a new log
I20260928 14:48:49.959636 29003 ts_tablet_manager.cc:1397] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:48:49.962775 29003 raft_consensus.cc:348] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.963047 29003 raft_consensus.cc:374] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:48:49.963083 29003 raft_consensus.cc:729] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 758645f666aa4a4e9719545817addc46, State: Initialized, Role: FOLLOWER
I20260928 14:48:49.963593 29003 consensus_queue.cc:260] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:49.964407 28988 heartbeater.cc:503] Master 127.13.223.126:35385 was elected leader, sending a full tablet report...
I20260928 14:48:49.964641 29003 ts_tablet_manager.cc:1428] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:48:50.089468 29006 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:50.089720 29006 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:50.091117 29006 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323)
I20260928 14:48:50.096048 28998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.091406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:59271 (local address 127.13.223.66:37623)
0928 14:48:50.091548 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:50.091555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:50.091574 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:48:50.091703 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:50.091707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:50.091767 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:50.091838 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:50.092312 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.093056 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.094612 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.095298 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.095476 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:50.095580 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:50.095788 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 14:48:50.095897 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:48:50.096094 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.091288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.13.223.67:59271)
0928 14:48:50.091436 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.091453 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:50.091545 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:50.091975 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:50.091977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:50.092020 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:50.092154 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:50.092164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.093206 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.093209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:50.094425 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:50.094436 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.095481 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.095486 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:50.095614 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:50.095643 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:48:50.095745 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 14:48:50.095855 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":107}
I20260928 14:48:50.096488 29007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.091406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.13.223.67:54115)
0928 14:48:50.091830 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.091847 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:50.091942 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:50.092313 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:50.092316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:50.092338 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:48:50.092463 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:50.092470 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.093370 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.093373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:50.094865 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:50.094899 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.095894 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.095898 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:50.096021 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:50.096139 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 14:48:50.096227 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 14:48:50.096317 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":27,"thread_start_us":123,"threads_started":1}
I20260928 14:48:50.096834 29008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.091585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:54115 (local address 127.13.223.65:33323)
0928 14:48:50.091878 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:50.091882 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:50.091901 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:48:50.092063 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:50.092066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:50.092119 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:50.092214 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:48:50.092615 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.093214 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.095120 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.095757 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.095886 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:50.095939 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:50.096565 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 14:48:50.096686 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":81,"threads_started":1}
I20260928 14:48:50.097118 28810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761af7ac93c94eb9b80e421c71fb8878" candidate_uuid: "758645f666aa4a4e9719545817addc46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" is_pre_election: true
I20260928 14:48:50.097507 28810 raft_consensus.cc:2454] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758645f666aa4a4e9719545817addc46 in term 0.
I20260928 14:48:50.097633 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761af7ac93c94eb9b80e421c71fb8878" candidate_uuid: "758645f666aa4a4e9719545817addc46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa3a640bee4f4e817fe247cb2754eb" is_pre_election: true
I20260928 14:48:50.097970 28653 raft_consensus.cc:2454] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 758645f666aa4a4e9719545817addc46 in term 0.
I20260928 14:48:50.098076 28876 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c; no voters:
I20260928 14:48:50.098380 29006 raft_consensus.cc:2781] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:48:50.098474 29006 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:48:50.098587 29006 raft_consensus.cc:3037] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:50.099736 29006 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:50.100222 29006 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [CANDIDATE]: Term 1 election: Requested vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323)
I20260928 14:48:50.100571 28810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761af7ac93c94eb9b80e421c71fb8878" candidate_uuid: "758645f666aa4a4e9719545817addc46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
I20260928 14:48:50.100752 28810 raft_consensus.cc:3037] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:50.100749 28653 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "761af7ac93c94eb9b80e421c71fb8878" candidate_uuid: "758645f666aa4a4e9719545817addc46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52fa3a640bee4f4e817fe247cb2754eb"
I20260928 14:48:50.100924 28653 raft_consensus.cc:3037] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:48:50.101954 28810 raft_consensus.cc:2454] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758645f666aa4a4e9719545817addc46 in term 1.
I20260928 14:48:50.102118 28653 raft_consensus.cc:2454] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 758645f666aa4a4e9719545817addc46 in term 1.
I20260928 14:48:50.102303 28876 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c; no voters:
I20260928 14:48:50.102613 29006 raft_consensus.cc:2781] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:48:50.102892 29006 raft_consensus.cc:686] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 1 LEADER]: Becoming Leader. State: Replica: 758645f666aa4a4e9719545817addc46, State: Running, Role: LEADER
I20260928 14:48:50.103399 29006 consensus_queue.cc:237] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:50.106952 28529 catalog_manager.cc:5495] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 758645f666aa4a4e9719545817addc46 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "758645f666aa4a4e9719545817addc46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } health_report { overall_health: UNKNOWN } } }
W20260928 14:48:50.128118 28989 tablet.cc:2369] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:50.137620 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.133614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.13.223.67:40455)
0928 14:48:50.133772 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:50.133776 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:50.133791 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:48:50.133902 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:50.133904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:50.133959 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:50.134060 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:48:50.134378 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.135063 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.135977 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:50.136202 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:50.136285 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:50.136616 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:48:50.137155 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:48:50.137262 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:50.137360 (+ 98us) server_negotiation.cc:299] Negotiation successful
0928 14:48:50.137438 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:48:50.144707 28810 raft_consensus.cc:1261] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Refusing update from remote peer 758645f666aa4a4e9719545817addc46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:50.144735 28653 raft_consensus.cc:1261] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Refusing update from remote peer 758645f666aa4a4e9719545817addc46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:48:50.145519 29006 consensus_queue.cc:1035] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:48:50.145735 29009 consensus_queue.cc:1035] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 14:48:50.153792 28718 tablet.cc:2369] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:48:50.157224 29013 mvcc.cc:205] Tried to move back new op lower bound from 7334325985863446528 to 7334325985706610688. Current Snapshot: MvccSnapshot[applied={T|T < 7334325985863446528}]
I20260928 14:48:50.159054 29012 mvcc.cc:205] Tried to move back new op lower bound from 7334325985863446528 to 7334325985706610688. Current Snapshot: MvccSnapshot[applied={T|T < 7334325985863446528 or (T in {7334325985863446528})}]
I20260928 14:48:50.165457 28528 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:57148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:50.165686 28528 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:57148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:48:50.168591 28528 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Sending DeleteTablet for 3 replicas of tablet 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:50.169469 28790 tablet_service.cc:1514] Processing DeleteTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:56486
I20260928 14:48:50.169520 28651 tablet_service.cc:1514] Processing DeleteTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:39114
I20260928 14:48:50.169643 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28567
I20260928 14:48:50.170202 28922 tablet_service.cc:1514] Processing DeleteTablet for tablet 761af7ac93c94eb9b80e421c71fb8878 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:48:50 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:53394
I20260928 14:48:50.170771 29021 tablet_replica.cc:330] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: stopping tablet replica
I20260928 14:48:50.170771 29022 tablet_replica.cc:330] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: stopping tablet replica
I20260928 14:48:50.171121 29022 raft_consensus.cc:2229] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:48:50.171114 29021 raft_consensus.cc:2229] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:48:50.171461 29021 raft_consensus.cc:2258] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:50.171613 29022 raft_consensus.cc:2258] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:48:50.174371 29022 ts_tablet_manager.cc:1905] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:50.174875 29021 ts_tablet_manager.cc:1905] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:50.177901 29021 ts_tablet_manager.cc:1918] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:50.178025 29021 log.cc:1196] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/761af7ac93c94eb9b80e421c71fb8878
W20260928 14:48:50.178228 28515 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:33323 (error 108)
I20260928 14:48:50.178493 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28721
I20260928 14:48:50.179491 29022 ts_tablet_manager.cc:1918] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:50.179700 29022 log.cc:1196] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:50.180141 29022 ts_tablet_manager.cc:1939] T 761af7ac93c94eb9b80e421c71fb8878 P 758645f666aa4a4e9719545817addc46: Deleting consensus metadata
I20260928 14:48:50.181089 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.180769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:33323 (local address 127.0.0.1:39116)
0928 14:48:50.180917 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.180994 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260928 14:48:50.181311 28515 catalog_manager.cc:4584] TS 52fa3a640bee4f4e817fe247cb2754eb (127.13.223.65:33323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 761af7ac93c94eb9b80e421c71fb8878: Network error: Client connection negotiation failed: client connection to 127.13.223.65:33323: connect: Connection refused (error 111)
I20260928 14:48:50.182308 28515 catalog_manager.cc:4841] TS 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): tablet 761af7ac93c94eb9b80e421c71fb8878 (table test-table [id=f7b445e9a3d8408797fc293b917180ac]) successfully deleted
I20260928 14:48:50.186619 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28859
I20260928 14:48:50.186873 28997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.186535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.0.0.1:56502)
0928 14:48:50.186709 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.186775 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:37623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 14:48:50.192543 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 28496
I20260928 14:48:50.200065 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--webserver_interface=127.13.223.126
--webserver_port=43549
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:35385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:50.325785 29024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:50.326104 29024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:50.326148 29024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:50.330507 29024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:48:50.330613 29024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:50.330634 29024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:50.330651 29024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:48:50.330667 29024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:48:50.336328 29024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:35385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=43549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:50.337651 29024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:50.338749 29024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:50.348459 29030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:50.348474 29032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.348529 29024 server_base.cc:1034] running on GCE node
W20260928 14:48:50.348474 29029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.349574 29024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:50.351042 29024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:50.352260 29024 hybrid_clock.cc:648] HybridClock initialized: now 1790606930352219 us; error 56 us; skew 500 ppm
Sep 28 14:48:50 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:50.355634 29024 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:48:50.356735 29024 webserver.cc:460] Webserver started at http://127.13.223.126:43549/ using document root <none> and password file <none>
I20260928 14:48:50.357402 29024 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:50.357473 29024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:50.361786 29024 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260928 14:48:50.364459 29039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:50.366058 29024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260928 14:48:50.366191 29024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c4121fe0d80648a9bfe7ed669dc28f01"
format_stamp: "Formatted at 2026-09-28 14:48:47 on dist-test-slave-gkhx"
server_key: "2e8aec4be54d39da0ce725ebdd7a85f0"
server_key_iv: "ee5ffb0c23fbe5de459ec82c507df686"
server_key_version: "encryptionkey@0"
I20260928 14:48:50.366886 29024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:50.395061 29024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:50.398893 29024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:50.399096 29024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:50.407397 29024 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:35385
I20260928 14:48:50.407807 29091 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:35385 every 8 connection(s)
I20260928 14:48:50.408736 29024 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:48:50.413617 29092 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:48:50.415572 29092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap starting.
I20260928 14:48:50.418488 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29024
I20260928 14:48:50.419092 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:33323
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=33747
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:50.436623 29092 log.cc:824] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:50.448800 29092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:50.449342 29092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01: Bootstrap complete.
I20260928 14:48:50.453804 29092 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:50.454310 29092 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Initialized, Role: FOLLOWER
I20260928 14:48:50.454982 29092 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:50.455160 29092 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:48:50.455260 29092 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:48:50.455382 29092 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:48:50.457511 29092 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:50.457911 29092 leader_election.cc:302] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4121fe0d80648a9bfe7ed669dc28f01; no voters:
I20260928 14:48:50.458323 29092 leader_election.cc:288] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:48:50.458424 29096 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:48:50.458662 29096 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [term 3 LEADER]: Becoming Leader. State: Replica: c4121fe0d80648a9bfe7ed669dc28f01, State: Running, Role: LEADER
I20260928 14:48:50.458984 29096 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } }
I20260928 14:48:50.459739 29092 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:48:50.460424 29097 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:50.460563 29097 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:50.461351 29098 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4121fe0d80648a9bfe7ed669dc28f01. Latest consensus state: current_term: 3 leader_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4121fe0d80648a9bfe7ed669dc28f01" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 35385 } } }
I20260928 14:48:50.461447 29098 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4121fe0d80648a9bfe7ed669dc28f01 [sys.catalog]: This master's current role is: LEADER
I20260928 14:48:50.461710 29103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:48:50.466466 29103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:48:50.467147 29103 catalog_manager.cc:1259] Loaded cluster ID: 20e4a720712a451bbdf1dafd64e51736
I20260928 14:48:50.467240 29103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 14:48:50.468204 29103 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5056a5a9ad google::LogMessage::Fail() at ??:0
@ 0x7f5056a5d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f5056a5a4de google::LogMessage::Flush() at ??:0
@ 0x7f5056a5ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f506139dfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f506139ed2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f506139b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f50613df640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5058cac3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5058cacc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5058cae3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5058c9d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f505a2e16db start_thread at ??:0
@ 0x7f505555861f clone at ??:0
W20260928 14:48:50.579206 29094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:50.579643 29094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:50.579715 29094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:50.586082 29094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:50.586192 29094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:50.586288 29094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:48:50.593168 29094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:33323
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=33747
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:50.594501 29094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:50.595706 29094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:50.606422 29134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:50.606422 29133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.606775 29094 server_base.cc:1034] running on GCE node
W20260928 14:48:50.606884 29136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.607602 29094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:50.608445 29094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:50.609668 29094 hybrid_clock.cc:648] HybridClock initialized: now 1790606930609658 us; error 68 us; skew 500 ppm
Sep 28 14:48:50 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:50.613762 29094 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:48:50.615280 29094 webserver.cc:460] Webserver started at http://127.13.223.65:33747/ using document root <none> and password file <none>
I20260928 14:48:50.615932 29094 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:50.616058 29094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:50.620787 29094 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 14:48:50.623734 29143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:50.625517 29094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:50.625667 29094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "52fa3a640bee4f4e817fe247cb2754eb"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "f5280f674d5f1dd3ccc1feae894f496b"
server_key_iv: "5d708d6fc7c9e17caa91c96679668aa4"
server_key_version: "encryptionkey@0"
I20260928 14:48:50.626602 29094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:50.647795 29094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:50.651970 29094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:50.652194 29094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:50.652957 29094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:50.654788 29150 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:48:50.659862 29094 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:48:50.659953 29094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260928 14:48:50.660013 29094 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:48:50.661981 29094 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:48:50.662055 29094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 14:48:50.662324 29150 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap starting.
I20260928 14:48:50.673658 29094 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:33323
I20260928 14:48:50.675150 29094 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:48:50.675436 29257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:33323 every 8 connection(s)
I20260928 14:48:50.678989 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29094
I20260928 14:48:50.679497 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:37623
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=33961
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:48:50.685689 29150 log.cc:824] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Log is configured to *not* fsync() on all Append() calls
I20260928 14:48:50.696198 29150 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:48:50.696731 29150 tablet_bootstrap.cc:492] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Bootstrap complete.
I20260928 14:48:50.698146 29150 ts_tablet_manager.cc:1397] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent bootstrapping tablet: real 0.036s user 0.027s sys 0.004s
I20260928 14:48:50.701699 29150 raft_consensus.cc:348] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:50.702524 29150 raft_consensus.cc:729] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52fa3a640bee4f4e817fe247cb2754eb, State: Initialized, Role: FOLLOWER
I20260928 14:48:50.703158 29150 consensus_queue.cc:260] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:50.704500 29150 ts_tablet_manager.cc:1428] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb: Time spent starting tablet: real 0.006s user 0.005s sys 0.004s
W20260928 14:48:50.725603 29260 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:48:50.677804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:48973)
0928 14:48:50.680125 (+ 2321us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.680215 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:50.681442 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:50.724774 (+ 43332us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2010,"thread_start_us":162,"threads_started":1}
W20260928 14:48:50.726423 29258 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:35385 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:35385: Client connection negotiation failed: client connection to 127.13.223.126:35385: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:48:50.727361 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.727055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:47689)
0928 14:48:50.727216 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.727292 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260928 14:48:50.728324 29260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.728097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:54163)
0928 14:48:50.728220 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.728269 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 14:48:50.728629 29258 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:48:50.812740 29261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:50.813040 29261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:50.813086 29261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:50.817309 29261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:50.817394 29261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:50.817456 29261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:48:50.823766 29261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:37623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=33961
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:50.825037 29261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:50.826261 29261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:50.837522 29269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.837706 29261 server_base.cc:1034] running on GCE node
W20260928 14:48:50.837505 29268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:50.837507 29271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:50.838366 29261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:50.839147 29261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:50.840348 29261 hybrid_clock.cc:648] HybridClock initialized: now 1790606930840298 us; error 68 us; skew 500 ppm
Sep 28 14:48:50 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:50.843559 29261 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:48:50.844583 29261 webserver.cc:460] Webserver started at http://127.13.223.66:33961/ using document root <none> and password file <none>
I20260928 14:48:50.845221 29261 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:50.845325 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:50.849606 29261 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:48:50.852186 29278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:50.853514 29261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:48:50.853665 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "76cab114e7ed4ff6b74565c0c69bcc5c"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "86bb093d0142ee94b984fb786c14a2a0"
server_key_iv: "2613fbe5d0c3c595e4f8e993334acd1d"
server_key_version: "encryptionkey@0"
I20260928 14:48:50.854298 29261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:50.896234 29261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:50.900272 29261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:50.900543 29261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:50.901275 29261 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:50.902684 29285 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:48:50.906919 29285 ts_tablet_manager.cc:1852] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:48:50.907352 29285 ts_tablet_manager.cc:1905] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:48:50.912474 29285 ts_tablet_manager.cc:1918] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:48:50.912601 29285 log.cc:1196] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:50.912989 29285 ts_tablet_manager.cc:1939] T 761af7ac93c94eb9b80e421c71fb8878 P 76cab114e7ed4ff6b74565c0c69bcc5c: Deleting consensus metadata
I20260928 14:48:50.913367 29261 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:48:50.913472 29261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.000s
I20260928 14:48:50.913559 29261 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:50.913625 29261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:50.924423 29261 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:37623
I20260928 14:48:50.924470 29392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:37623 every 8 connection(s)
I20260928 14:48:50.925403 29261 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:48:50.929555 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.928103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:53775)
0928 14:48:50.928654 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.928776 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":126,"threads_started":1}
W20260928 14:48:50.929984 29393 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:35385 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:35385: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
I20260928 14:48:50.930733 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.930410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:54745)
0928 14:48:50.930570 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.930666 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:48:50.931315 29395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:50.931120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:55391)
0928 14:48:50.931220 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:50.931261 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:48:50.931592 29393 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:48:50.932811 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29261
I20260928 14:48:50.933254 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:40455
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:35385
--webserver_port=40731
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:40193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:48:51.055495 29396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:48:51.055835 29396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:48:51.055882 29396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:48:51.060734 29396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:48:51.060820 29396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:48:51.060890 29396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:48:51.066825 29396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:40193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:40455
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=40731
--tserver_master_addrs=127.13.223.126:35385
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:48:51.068058 29396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:48:51.069154 29396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:48:51.079718 29401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:51.079696 29404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:48:51.079696 29402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:48:51.079777 29396 server_base.cc:1034] running on GCE node
I20260928 14:48:51.080387 29396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:48:51.081172 29396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:48:51.082388 29396 hybrid_clock.cc:648] HybridClock initialized: now 1790606931082359 us; error 45 us; skew 500 ppm
Sep 28 14:48:51 dist-test-slave-gkhx krb5kdc[27947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606931, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:48:51.085559 29396 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:48:51.086714 29396 webserver.cc:460] Webserver started at http://127.13.223.67:40731/ using document root <none> and password file <none>
I20260928 14:48:51.087356 29396 fs_manager.cc:359] Metadata directory not provided
I20260928 14:48:51.087466 29396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:48:51.091936 29396 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:48:51.094465 29411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:51.096086 29396 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:48:51.096311 29396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "758645f666aa4a4e9719545817addc46"
format_stamp: "Formatted at 2026-09-28 14:48:48 on dist-test-slave-gkhx"
server_key: "b1017585b7c18a20e6a03268a3a7e3fb"
server_key_iv: "5cf606878c6ca7aca9d3208d437f87a2"
server_key_version: "encryptionkey@0"
I20260928 14:48:51.096961 29396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:48:51.112337 29396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:48:51.116176 29396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:48:51.116416 29396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:48:51.117146 29396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:48:51.118129 29396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:48:51.118221 29396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:51.118299 29396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:48:51.118336 29396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:48:51.129730 29396 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40455
I20260928 14:48:51.129750 29524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40455 every 8 connection(s)
I20260928 14:48:51.131035 29396 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:48:51.131278 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29396
I20260928 14:48:51.136111 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.134179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:34913)
0928 14:48:51.134762 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.135253 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":125,"threads_started":1}
W20260928 14:48:51.136710 29525 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:35385 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:35385: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
I20260928 14:48:51.137530 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.137223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:50899)
0928 14:48:51.137393 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.137457 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 14:48:51.138360 29528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.138129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:54741)
0928 14:48:51.138249 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.138307 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 14:48:51.138697 29525 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:48:51.729760 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.729282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:48087)
0928 14:48:51.729569 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.729654 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":72,"threads_started":1}
I20260928 14:48:51.833304 29530 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:51.833500 29530 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:51.834925 29530 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
Sep 28 14:48:51 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.66@KRBTEST.COM
Sep 28 14:48:51 dist-test-slave-gkhx krb5kdc[27947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606930, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.67@KRBTEST.COM
I20260928 14:48:51.858929 29532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.835580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:42987 (local address 127.13.223.67:40455)
0928 14:48:51.836010 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:51.836026 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:51.836200 (+ 174us) server_negotiation.cc:407] Connection header received
0928 14:48:51.836913 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:51.836920 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:51.837284 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:51.837723 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:51.839285 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.840628 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842089 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.842670 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842856 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:51.853745 (+ 10889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:51.853751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:51.853765 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:51.853801 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:51.855870 (+ 2069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:51.856489 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:51.856502 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:51.856507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:51.856648 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:51.857044 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:51.857049 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:51.857051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:51.857599 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:51.857753 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:51.858354 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 14:48:51.858691 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":131,"threads_started":1}
I20260928 14:48:51.858914 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.835301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40455 (local address 127.13.223.65:42987)
0928 14:48:51.836048 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.836065 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:51.836197 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:51.837924 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:51.837927 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:51.838334 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:51.839039 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:51.839052 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.840818 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.840824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:51.841929 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:51.841940 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.842807 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:51.843054 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:51.850933 (+ 7879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:51.850973 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:51.853481 (+ 2508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:51.856023 (+ 2542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:51.856029 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:51.856032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:51.856337 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:51.856761 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:51.856765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:51.856768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:51.856932 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:51.857726 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:51.857733 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:51.858010 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 14:48:51.858224 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 14:48:51.858702 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":631,"spinlock_wait_cycles":58240,"thread_start_us":60,"threads_started":1}
I20260928 14:48:51.859031 29529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.835132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:37623 (local address 127.13.223.65:36005)
0928 14:48:51.835768 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.835787 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:48:51.835918 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:48:51.837728 (+ 1810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:48:51.837734 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:48:51.838349 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:48:51.839031 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:51.839050 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.840811 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.840820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:51.841894 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:48:51.841905 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.842787 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842791 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:48:51.843049 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:51.850933 (+ 7884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:48:51.850968 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:48:51.853132 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:48:51.855924 (+ 2792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:51.855932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:51.855951 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:51.856334 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:51.856799 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:48:51.856803 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:48:51.856805 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:48:51.856941 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:48:51.857726 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:48:51.857734 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:48:51.858011 (+ 277us) client_negotiation.cc:769] Sending connection context
0928 14:48:51.858226 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 14:48:51.858810 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":454,"spinlock_wait_cycles":38784}
I20260928 14:48:51.859897 29531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.835310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:36005 (local address 127.13.223.66:37623)
0928 14:48:51.835740 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0928 14:48:51.835757 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:48:51.835993 (+ 236us) server_negotiation.cc:407] Connection header received
0928 14:48:51.836855 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:48:51.836863 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:48:51.837284 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:48:51.837658 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:48:51.839249 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.840606 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842042 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:48:51.842640 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:48:51.842842 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:48:51.853495 (+ 10653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:48:51.853501 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:48:51.853517 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:48:51.853555 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:48:51.855729 (+ 2174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:51.856523 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:51.856536 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:51.856541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:51.856679 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:48:51.857062 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:48:51.857067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:48:51.857069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:48:51.857582 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:48:51.857802 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0928 14:48:51.858367 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 14:48:51.858729 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":176,"threads_started":1}
W20260928 14:48:51.862320 29146 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
W20260928 14:48:51.862524 29145 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:51.862684 29145 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:48:51.862974 29530 raft_consensus.cc:2726] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:48:51.932817 29534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:51.932378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:44835)
0928 14:48:51.932650 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:51.932715 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":69,"threads_started":1}
I20260928 14:48:52.139851 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:52.139350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:35577)
0928 14:48:52.139673 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:52.139737 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":136,"threads_started":1}
I20260928 14:48:52.731496 29536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:52.730952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:49391)
0928 14:48:52.731264 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:52.731333 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":92,"threads_started":1}
I20260928 14:48:52.934345 29537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:52.933827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:51695)
0928 14:48:52.934170 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:52.934243 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":78,"threads_started":1}
I20260928 14:48:53.141362 29538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:53.140853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:46673)
0928 14:48:53.141151 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:53.141216 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":93,"threads_started":1}
I20260928 14:48:53.444883 29539 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:53.445148 29539 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:53.445701 29539 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
W20260928 14:48:53.446678 29145 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
W20260928 14:48:53.446797 29146 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:53.446909 29146 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:48:53.447166 29539 raft_consensus.cc:2726] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:48:53.733033 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:53.732539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:59905)
0928 14:48:53.732855 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:53.732926 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":119,"threads_started":1}
I20260928 14:48:53.935920 29541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:53.935430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:38639)
0928 14:48:53.935744 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:53.935815 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":96,"threads_started":1}
I20260928 14:48:54.143518 29542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:54.142844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:51243)
0928 14:48:54.143285 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:54.143396 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":120,"threads_started":1}
I20260928 14:48:54.734709 29543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:54.734110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:39925)
0928 14:48:54.734423 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:54.734486 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":95,"threads_started":1}
I20260928 14:48:54.937315 29544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:54.936798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:36623)
0928 14:48:54.937097 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:54.937212 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":90,"threads_started":1}
I20260928 14:48:55.145220 29545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:55.144648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:60317)
0928 14:48:55.144989 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:55.145106 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":100,"threads_started":1}
I20260928 14:48:55.736485 29546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:55.735941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:46813)
0928 14:48:55.736315 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:55.736384 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":92,"threads_started":1}
I20260928 14:48:55.812402 29547 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:55.812559 29547 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:55.813223 29547 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
W20260928 14:48:55.814134 29146 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
W20260928 14:48:55.814263 29145 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:55.814399 29145 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:48:55.814754 29547 raft_consensus.cc:2726] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:48:55.939447 29548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:55.938939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:56299)
0928 14:48:55.939284 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:55.939347 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":137,"threads_started":1}
I20260928 14:48:56.146586 29549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:56.146095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:56443)
0928 14:48:56.146396 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:56.146454 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260928 14:48:56.738101 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:56.737611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:36007)
0928 14:48:56.737924 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:56.737998 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":88,"threads_started":1}
I20260928 14:48:56.941044 29551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:56.940376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:39853)
0928 14:48:56.940792 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:56.940866 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":80,"threads_started":1}
I20260928 14:48:57.147996 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:57.147498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:42415)
0928 14:48:57.147818 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:57.147891 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260928 14:48:57.688340 29553 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:48:57.688484 29553 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:48:57.689016 29553 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
W20260928 14:48:57.690006 29145 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
W20260928 14:48:57.690146 29146 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:48:57.690250 29146 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:48:57.690472 29553 raft_consensus.cc:2726] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:48:57.739846 29554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:57.739236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:58055)
0928 14:48:57.739638 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:57.739725 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":101,"threads_started":1}
I20260928 14:48:57.942747 29555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:57.942124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:35009)
0928 14:48:57.942474 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:57.942599 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":100,"threads_started":1}
I20260928 14:48:58.149581 29556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:58.148935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:36743)
0928 14:48:58.149403 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:58.149474 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":98,"threads_started":1}
I20260928 14:48:58.741324 29557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:58.740820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:58441)
0928 14:48:58.741157 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:58.741225 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260928 14:48:58.944303 29558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:58.943767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:46911)
0928 14:48:58.944115 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:58.944190 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":93,"threads_started":1}
I20260928 14:48:59.151253 29559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:59.150710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:60429)
0928 14:48:59.151055 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:59.151133 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":94,"threads_started":1}
I20260928 14:48:59.743192 29560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:59.742668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:44467)
0928 14:48:59.742957 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:59.743024 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":75,"threads_started":1}
I20260928 14:48:59.945799 29561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:48:59.945280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:39885)
0928 14:48:59.945597 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0928 14:48:59.945692 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":87,"threads_started":1}
I20260928 14:49:00.152750 29562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:00.152247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:36705)
0928 14:49:00.152581 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:00.152648 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":85,"threads_started":1}
I20260928 14:49:00.488632 29563 raft_consensus.cc:482] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:00.488806 29563 raft_consensus.cc:504] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76cab114e7ed4ff6b74565c0c69bcc5c" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 37623 } } peers { permanent_uuid: "758645f666aa4a4e9719545817addc46" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40455 } } peers { permanent_uuid: "52fa3a640bee4f4e817fe247cb2754eb" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 33323 } }
I20260928 14:49:00.489305 29563 leader_election.cc:288] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623), 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455)
W20260928 14:49:00.490304 29146 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 758645f666aa4a4e9719545817addc46 (127.13.223.67:40455): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
W20260928 14:49:00.490425 29145 leader_election.cc:341] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76cab114e7ed4ff6b74565c0c69bcc5c (127.13.223.66:37623): Not found: Tablet not found: 761af7ac93c94eb9b80e421c71fb8878
I20260928 14:49:00.490541 29145 leader_election.cc:302] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 52fa3a640bee4f4e817fe247cb2754eb; no voters: 758645f666aa4a4e9719545817addc46, 76cab114e7ed4ff6b74565c0c69bcc5c
I20260928 14:49:00.490784 29563 raft_consensus.cc:2726] T 761af7ac93c94eb9b80e421c71fb8878 P 52fa3a640bee4f4e817fe247cb2754eb [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:49:00.744913 29564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:00.744341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:56177)
0928 14:49:00.744726 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:00.744802 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":85,"threads_started":1}
I20260928 14:49:00.947575 29565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:00.947092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:43443)
0928 14:49:00.947403 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:00.947476 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":118,"threads_started":1}
I20260928 14:49:01.154383 29566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:01.153844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:45371)
0928 14:49:01.154195 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:01.154271 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":79,"threads_started":1}
I20260928 14:49:01.746534 29567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:01.745996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:44221)
0928 14:49:01.746319 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:01.746394 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":138,"threads_started":1}
I20260928 14:49:01.949199 29568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:01.948462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:42727)
0928 14:49:01.948797 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:01.948872 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":109,"threads_started":1}
I20260928 14:49:02.155990 29569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:02.155507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:37819)
0928 14:49:02.155793 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:02.155881 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":79,"threads_started":1}
I20260928 14:49:02.748101 29570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:02.747594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:50037)
0928 14:49:02.747946 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:02.748000 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260928 14:49:02.950675 29571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:02.950185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:58819)
0928 14:49:02.950457 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:02.950510 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":115,"threads_started":1}
I20260928 14:49:03.157734 29572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:03.157223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:47699)
0928 14:49:03.157539 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:03.157620 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":104,"threads_started":1}
I20260928 14:49:03.749689 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:03.749184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:34863)
0928 14:49:03.749474 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:03.749528 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":91,"threads_started":1}
I20260928 14:49:03.951994 29574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:03.951554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:44727)
0928 14:49:03.951836 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:03.951896 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":74,"threads_started":1}
I20260928 14:49:04.159152 29575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:04.158712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:54415)
0928 14:49:04.158977 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:04.159045 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260928 14:49:04.751389 29576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:04.750860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:41915)
0928 14:49:04.751157 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:04.751235 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":103,"threads_started":1}
I20260928 14:49:04.953619 29577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:04.953015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:33985)
0928 14:49:04.953382 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:04.953453 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":112,"threads_started":1}
I20260928 14:49:05.160741 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:05.160163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.67:33997)
0928 14:49:05.160465 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:05.160560 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260928 14:49:05.753067 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:05.752559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.65:56655)
0928 14:49:05.752894 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:05.752964 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":86,"threads_started":1}
I20260928 14:49:05.955343 29580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:05.954829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:35385 (local address 127.13.223.66:52821)
0928 14:49:05.955100 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:05.955242 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260928 14:49:06.139546 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29094
I20260928 14:49:06.145781 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29261
I20260928 14:49:06.152310 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29396
2026-09-28T14:49:06Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19800 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:49:06 dist-test-slave-gkhx krb5kdc[29583](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:49:06 dist-test-slave-gkhx krb5kdc[29583](info): set up 2 sockets
Sep 28 14:49:06 dist-test-slave-gkhx krb5kdc[29583](info): commencing operation
krb5kdc: starting...
W20260928 14:49:07.241575 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:49:07 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:49:07.386799 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:07.513743 29596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:07.514082 29596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:07.514161 29596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:07.518404 29596 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:49:07.518510 29596 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:49:07.518536 29596 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:49:07.518559 29596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:49:07.518615 29596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:07.518653 29596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:07.518694 29596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:49:07.518749 29596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:49:07.524807 29596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:07.526096 29596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:07.527216 29596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:07.536777 29601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:07.536907 29596 server_base.cc:1034] running on GCE node
W20260928 14:49:07.536760 29604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:07.536760 29602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:07.537871 29596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:07.539081 29596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:07.540272 29596 hybrid_clock.cc:648] HybridClock initialized: now 1790606947540258 us; error 28 us; skew 500 ppm
Sep 28 14:49:07 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:07.543603 29596 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:49:07.544713 29596 webserver.cc:460] Webserver started at http://127.13.223.126:44253/ using document root <none> and password file <none>
I20260928 14:49:07.545351 29596 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:07.545454 29596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:07.545723 29596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:49:07.548116 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "a2e6a68d140e4cdb8870c95f6fed42f5"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "4a9d277bcea8223da96b0568d25a810b"
server_key_iv: "00667cb260effcb0dc2b326ad41b633a"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.548816 29596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "a2e6a68d140e4cdb8870c95f6fed42f5"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "4a9d277bcea8223da96b0568d25a810b"
server_key_iv: "00667cb260effcb0dc2b326ad41b633a"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.553351 29596 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:49:07.556463 29611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:07.557999 29596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:07.558173 29596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "a2e6a68d140e4cdb8870c95f6fed42f5"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "4a9d277bcea8223da96b0568d25a810b"
server_key_iv: "00667cb260effcb0dc2b326ad41b633a"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.558317 29596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:07.589541 29596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:07.593456 29596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:07.593729 29596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:07.602876 29596 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44297
I20260928 14:49:07.603175 29663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44297 every 8 connection(s)
I20260928 14:49:07.604151 29596 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:49:07.605314 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29596
I20260928 14:49:07.605434 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:49:07.605800 14205 external_mini_cluster.cc:1428] Setting key 60b70d51e482081783412f42f870ab21
I20260928 14:49:07.607903 29664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:07.614619 29664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap starting.
Sep 28 14:49:07 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:07.617620 29664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:07.618678 29664 log.cc:824] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:07.621409 29664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: No bootstrap required, opened a new log
I20260928 14:49:07.622759 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:07.607368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38290 (local address 127.13.223.126:44297)
0928 14:49:07.607887 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:07.607909 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:07.608049 (+ 140us) server_negotiation.cc:407] Connection header received
0928 14:49:07.608934 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:07.608951 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:07.609326 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:07.609732 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:07.610703 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.611881 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.612891 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.613451 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.613653 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:07.616896 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:07.616901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:07.616917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:07.616952 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:07.619209 (+ 2257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:07.619889 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:07.619894 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:07.619899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:07.620016 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:07.620347 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:07.620349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:07.620350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:07.620797 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:07.620956 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:07.621372 (+ 416us) server_negotiation.cc:299] Negotiation successful
0928 14:49:07.621753 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":143,"threads_started":1}
I20260928 14:49:07.625730 29664 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:07.625984 29664 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:07.626084 29664 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Initialized, Role: FOLLOWER
I20260928 14:49:07.626601 29664 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:07.626760 29664 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:49:07.626884 29664 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:49:07.627031 29664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:07.628110 29664 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:07.628497 29664 leader_election.cc:302] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e6a68d140e4cdb8870c95f6fed42f5; no voters:
I20260928 14:49:07.628856 29664 leader_election.cc:288] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:49:07.629040 29669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:49:07.629311 29669 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 LEADER]: Becoming Leader. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Running, Role: LEADER
I20260928 14:49:07.629750 29669 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:07.630301 29664 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:49:07.631765 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:07.631770 29671 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e6a68d140e4cdb8870c95f6fed42f5. Latest consensus state: current_term: 1 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:07.631928 29671 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:07.631929 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:07.632437 29678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:49:07.636121 29678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:49:07.642612 29678 catalog_manager.cc:1347] Generated new cluster ID: bf063a928547454fb23526aea2eb0ba0
I20260928 14:49:07.642717 29678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:49:07.667578 29678 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:49:07.668602 29678 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:49:07.674974 29678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Generated new TSK 0
I20260928 14:49:07.675774 29678 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:49:07.719810 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:07.862226 29690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:07.862788 29690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:07.862903 29690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:07.868072 29690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:07.868203 29690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:07.868330 29690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:49:07.875022 29690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:07.876374 29690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:07.877661 29690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:07.889130 29696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:07.889108 29698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:07.889539 29695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:07.889753 29690 server_base.cc:1034] running on GCE node
I20260928 14:49:07.890254 29690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:07.891136 29690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:07.892356 29690 hybrid_clock.cc:648] HybridClock initialized: now 1790606947892330 us; error 39 us; skew 500 ppm
Sep 28 14:49:07 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:07.895924 29690 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:49:07.897102 29690 webserver.cc:460] Webserver started at http://127.13.223.65:42055/ using document root <none> and password file <none>
I20260928 14:49:07.897835 29690 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:07.897917 29690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:07.898176 29690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:49:07.900725 29690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "64b754c626274b11b521d4a74b28bdd9"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "cd3da4533281b6dbd65ccf3f735e6786"
server_key_iv: "c11b8efe249132a60001427bf2c38ba8"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.901400 29690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "64b754c626274b11b521d4a74b28bdd9"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "cd3da4533281b6dbd65ccf3f735e6786"
server_key_iv: "c11b8efe249132a60001427bf2c38ba8"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.905930 29690 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:49:07.909175 29705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:07.911024 29690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:07.911178 29690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "64b754c626274b11b521d4a74b28bdd9"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "cd3da4533281b6dbd65ccf3f735e6786"
server_key_iv: "c11b8efe249132a60001427bf2c38ba8"
server_key_version: "encryptionkey@0"
I20260928 14:49:07.911329 29690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:07.925539 29690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:07.929884 29690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:07.930173 29690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:07.931020 29690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:07.932155 29690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:07.932214 29690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:07.932265 29690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:07.932327 29690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:07.943722 29690 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:44013
I20260928 14:49:07.943753 29818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:44013 every 8 connection(s)
I20260928 14:49:07.944854 29690 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:49:07.949061 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29690
I20260928 14:49:07.949190 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:49:07.949529 14205 external_mini_cluster.cc:1428] Setting key e7178e7918ab9cf1fc76e51559744dac
Sep 28 14:49:07 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606947, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:49:07.961763 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:07.947264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:60359 (local address 127.13.223.126:44297)
0928 14:49:07.947441 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:07.947446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:07.948352 (+ 906us) server_negotiation.cc:407] Connection header received
0928 14:49:07.949867 (+ 1515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:07.949870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:07.949931 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:07.950043 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:07.951543 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.952097 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.953326 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.953571 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.953691 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:07.956987 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:07.956993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:07.957000 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:07.957039 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:07.958964 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:07.959690 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:07.959694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:07.959696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:07.959794 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:07.960336 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:07.960342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:07.960347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:07.960575 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:07.960659 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:07.961407 (+ 748us) server_negotiation.cc:299] Negotiation successful
0928 14:49:07.961555 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:49:07.963078 29821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:07.947650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:60359)
0928 14:49:07.948176 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:07.948224 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:07.949618 (+ 1394us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:07.950244 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:07.950272 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:07.950851 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:07.951393 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:07.951407 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.952235 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.952239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:07.953179 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:07.953189 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:07.953739 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:07.953743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:07.953936 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:07.954685 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:07.954712 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:07.956756 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:07.959152 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:07.959159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:07.959185 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:07.959559 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:07.959907 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:07.959909 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:07.959911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:07.960222 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:07.960719 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:07.960725 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:07.961109 (+ 384us) client_negotiation.cc:769] Sending connection context
0928 14:49:07.961353 (+ 244us) client_negotiation.cc:240] Negotiation successful
0928 14:49:07.961760 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":122,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:49:07.964649 29819 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:07.964949 29819 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:07.965703 29819 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
I20260928 14:49:07.967855 29624 ts_manager.cc:194] Registered new tserver with Master: 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013)
I20260928 14:49:07.969197 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:60359
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:49:07.982931 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:08.102993 29824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:08.103365 29824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:08.103443 29824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:08.107900 29824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:08.108019 29824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:08.108124 29824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:49:08.113940 29824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:08.115294 29824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:08.116464 29824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:08.127345 29832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:08.127355 29829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:08.127345 29830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:08.127818 29824 server_base.cc:1034] running on GCE node
I20260928 14:49:08.128371 29824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:08.129148 29824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:08.130354 29824 hybrid_clock.cc:648] HybridClock initialized: now 1790606948130327 us; error 41 us; skew 500 ppm
Sep 28 14:49:08 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:08.133982 29824 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:49:08.135076 29824 webserver.cc:460] Webserver started at http://127.13.223.66:33797/ using document root <none> and password file <none>
I20260928 14:49:08.135660 29824 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:08.135731 29824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:08.135907 29824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:49:08.138088 29824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "a837633abfca4451b3e3159169b08f7b"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "2562ba80004241680e2a623bfd350172"
server_key_iv: "4e743620ec14604a7429e240320de3ba"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.138749 29824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "a837633abfca4451b3e3159169b08f7b"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "2562ba80004241680e2a623bfd350172"
server_key_iv: "4e743620ec14604a7429e240320de3ba"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.142935 29824 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:49:08.145957 29839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.147717 29824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:08.147854 29824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "a837633abfca4451b3e3159169b08f7b"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "2562ba80004241680e2a623bfd350172"
server_key_iv: "4e743620ec14604a7429e240320de3ba"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.148130 29824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:08.163079 29824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:08.167080 29824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:08.167285 29824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:08.167932 29824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:08.168885 29824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:08.168948 29824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.168996 29824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:08.169013 29824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.179908 29824 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35543
I20260928 14:49:08.179927 29952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35543 every 8 connection(s)
I20260928 14:49:08.181042 29824 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:49:08.190713 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29824
I20260928 14:49:08.190910 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:49:08.191320 14205 external_mini_cluster.cc:1428] Setting key 0f4890aa2a686b4224004811d71f2b58
Sep 28 14:49:08 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:08.198341 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.183302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:49721 (local address 127.13.223.126:44297)
0928 14:49:08.183456 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.183460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.184450 (+ 990us) server_negotiation.cc:407] Connection header received
0928 14:49:08.185796 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.185800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.185856 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.186041 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:08.187454 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.188148 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.189400 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.189694 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.189794 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.193097 (+ 3303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:08.193101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:08.193106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:08.193150 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:08.195269 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.196149 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.196153 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.196154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.196244 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.196722 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.196725 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.196726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.197026 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:08.197122 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.197993 (+ 871us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.198144 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:49:08.199646 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.183751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:49721)
0928 14:49:08.184265 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.184315 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.185538 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.186229 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.186236 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.186738 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:08.187275 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.187290 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.188295 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.188297 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.189245 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.189256 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.189855 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.189857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.190018 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.190840 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:08.190858 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:08.192782 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:08.195456 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:08.195467 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:08.195490 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:08.195939 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:08.196355 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:08.196360 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:08.196362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:08.196590 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:08.197169 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:08.197179 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:08.197581 (+ 402us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.197818 (+ 237us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.198262 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260928 14:49:08.201222 29953 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:08.201534 29953 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:08.202239 29953 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:49:08.203734 29624 ts_manager.cc:194] Registered new tserver with Master: a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543)
I20260928 14:49:08.204541 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:49721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:49:08.227387 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:08.355265 29958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:08.355571 29958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:08.355621 29958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:08.359839 29958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:08.359908 29958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:08.359968 29958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:49:08.365743 29958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:08.366839 29958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:08.367851 29958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:08.378438 29964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:08.378468 29963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:08.378484 29958 server_base.cc:1034] running on GCE node
W20260928 14:49:08.378438 29966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:08.379266 29958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:08.380025 29958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:08.381237 29958 hybrid_clock.cc:648] HybridClock initialized: now 1790606948381200 us; error 53 us; skew 500 ppm
Sep 28 14:49:08 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:08.384384 29958 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:49:08.385438 29958 webserver.cc:460] Webserver started at http://127.13.223.67:46705/ using document root <none> and password file <none>
I20260928 14:49:08.386046 29958 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:08.386155 29958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:08.386412 29958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:49:08.388659 29958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "e2558fec68df43db95e65d11d52132e4"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "dd46f7e90beaa60be05ecb4943715ebf"
server_key_iv: "e26f0e5ccaf2c26b62a6f92256862a63"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.389271 29958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "e2558fec68df43db95e65d11d52132e4"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "dd46f7e90beaa60be05ecb4943715ebf"
server_key_iv: "e26f0e5ccaf2c26b62a6f92256862a63"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.393484 29958 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:49:08.396607 29973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.398025 29958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:08.398229 29958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "e2558fec68df43db95e65d11d52132e4"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "dd46f7e90beaa60be05ecb4943715ebf"
server_key_iv: "e26f0e5ccaf2c26b62a6f92256862a63"
server_key_version: "encryptionkey@0"
I20260928 14:49:08.398371 29958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:08.413954 29958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:08.417830 29958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:08.418068 29958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:08.418776 29958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:08.419648 29958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:08.419720 29958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.419796 29958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:08.419839 29958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:08.429935 29958 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:41647
I20260928 14:49:08.430102 30086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:41647 every 8 connection(s)
I20260928 14:49:08.431036 29958 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:49:08.436101 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 29958
I20260928 14:49:08.436215 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:49:08.436494 14205 external_mini_cluster.cc:1428] Setting key f76cddc321c08c21ca74e163695b7495
Sep 28 14:49:08 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606948, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:08.446985 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.433381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:49525 (local address 127.13.223.126:44297)
0928 14:49:08.433545 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.433551 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.434674 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 14:49:08.435712 (+ 1038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.435717 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.435805 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.435896 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:08.437409 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.438131 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.439538 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.439793 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.439880 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.442816 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:08.442820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:08.442822 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:08.442852 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:08.444412 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.445062 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.445065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.445067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.445164 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.445627 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.445631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.445632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.445855 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:08.445977 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.446724 (+ 747us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.446848 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:49:08.447912 30089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.433706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:49525)
0928 14:49:08.434465 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.434509 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.435439 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.436118 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.436126 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.436614 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:08.437207 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.437240 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.438274 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.438278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.439402 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.439412 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.439931 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.439933 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.440079 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.440803 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:08.440831 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:08.442538 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:08.444553 (+ 2015us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:08.444558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:08.444572 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:08.444858 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:08.445306 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:08.445309 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:08.445311 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:08.445480 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:08.445997 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:08.446003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:08.446374 (+ 371us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.446642 (+ 268us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.446960 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":107,"threads_started":1}
I20260928 14:49:08.449324 30087 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:08.449626 30087 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:08.450304 30087 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
I20260928 14:49:08.451543 29624 ts_manager.cc:194] Registered new tserver with Master: e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
I20260928 14:49:08.452167 29624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:49525
I20260928 14:49:08.463677 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:49:08.475581 29667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.466285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.13.223.126:44297)
0928 14:49:08.466421 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.466425 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.466623 (+ 198us) server_negotiation.cc:407] Connection header received
0928 14:49:08.466771 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.466773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.466817 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.466895 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:08.467789 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.468413 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.469414 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.469643 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.469736 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.471504 (+ 1768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:08.471508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:08.471511 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:08.471542 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:08.473396 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.473956 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.473960 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.473962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.474055 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:08.474463 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:08.474465 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:08.474467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:08.474822 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:08.474946 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.475321 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.475447 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:49:08.479351 29624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38296:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:49:08.482934 29624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:49:08.500002 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.493293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59714 (local address 127.13.223.65:44013)
0928 14:49:08.493751 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.493759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.494241 (+ 482us) server_negotiation.cc:407] Connection header received
0928 14:49:08.494714 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.494721 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.494957 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.495135 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:08.495557 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.496526 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.498514 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.499285 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.499418 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.499601 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.499732 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.499834 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":86,"threads_started":1}
I20260928 14:49:08.500283 30100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.493876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:41647 (local address 127.0.0.1:60748)
0928 14:49:08.494640 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.494657 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.494749 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.495167 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.495173 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.495196 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:08.495350 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.495357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.496584 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.496591 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.498371 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.498383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.499638 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.499642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.499749 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.499975 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.500034 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.500125 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":519,"thread_start_us":90,"threads_started":1}
I20260928 14:49:08.500563 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.493525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35543 (local address 127.0.0.1:33464)
0928 14:49:08.494857 (+ 1332us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.494866 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.494971 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.496420 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.496423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.496440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:08.496521 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.496569 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.497783 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.497786 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.498783 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.498794 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.500152 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.500156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.500251 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.500272 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.500348 (+ 76us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.500422 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":971,"thread_start_us":102,"threads_started":1}
I20260928 14:49:08.500763 30098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.493125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44013 (local address 127.0.0.1:59714)
0928 14:49:08.494083 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.494142 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.494406 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.495307 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.495311 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.495330 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:08.495442 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.495448 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.496675 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.496678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.498371 (+ 1693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.498383 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.499417 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.499420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.499534 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.499572 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.499692 (+ 120us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.499830 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":95,"thread_start_us":106,"threads_started":1}
I20260928 14:49:08.501015 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.494096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60748 (local address 127.13.223.67:41647)
0928 14:49:08.494706 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.494717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.494738 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:49:08.494840 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.494844 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.494992 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.495161 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:08.495484 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.496278 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.498609 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.499461 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.500492 (+ 1031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.500688 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.500749 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.500863 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":80,"threads_started":1}
I20260928 14:49:08.501068 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.493944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.13.223.66:35543)
0928 14:49:08.494490 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.494504 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.494962 (+ 458us) server_negotiation.cc:407] Connection header received
0928 14:49:08.495131 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.495137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.495327 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.495496 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:08.496703 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.497636 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.498919 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.499806 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.499918 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.500087 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.500817 (+ 730us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.500910 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":148,"threads_started":1}
I20260928 14:49:08.503896 30021 tablet_service.cc:1467] Processing CreateTablet for tablet aeddb18b0edb454597676f3a3d72be82 (DEFAULT_TABLE table=test-table [id=23b63db092b3487c8d96193e468599d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:08.504019 29753 tablet_service.cc:1467] Processing CreateTablet for tablet aeddb18b0edb454597676f3a3d72be82 (DEFAULT_TABLE table=test-table [id=23b63db092b3487c8d96193e468599d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:08.505136 30021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeddb18b0edb454597676f3a3d72be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:08.504659 29887 tablet_service.cc:1467] Processing CreateTablet for tablet aeddb18b0edb454597676f3a3d72be82 (DEFAULT_TABLE table=test-table [id=23b63db092b3487c8d96193e468599d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:08.505170 29753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeddb18b0edb454597676f3a3d72be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:08.505760 29887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aeddb18b0edb454597676f3a3d72be82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:08.511873 30104 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Bootstrap starting.
I20260928 14:49:08.513751 30106 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Bootstrap starting.
I20260928 14:49:08.513751 30105 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap starting.
I20260928 14:49:08.514431 30104 tablet_bootstrap.cc:654] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:08.516084 30104 log.cc:824] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:08.516528 30105 tablet_bootstrap.cc:654] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:08.516983 30106 tablet_bootstrap.cc:654] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:08.517603 30105 log.cc:824] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:08.517885 30106 log.cc:824] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:08.518725 30104 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: No bootstrap required, opened a new log
I20260928 14:49:08.519021 30104 ts_tablet_manager.cc:1397] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:49:08.519886 30105 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: No bootstrap required, opened a new log
I20260928 14:49:08.520151 30105 ts_tablet_manager.cc:1397] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:49:08.520243 30106 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: No bootstrap required, opened a new log
I20260928 14:49:08.520454 30106 ts_tablet_manager.cc:1397] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 14:49:08.523914 30104 raft_consensus.cc:348] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.524355 30104 raft_consensus.cc:374] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:08.524433 30104 raft_consensus.cc:729] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2558fec68df43db95e65d11d52132e4, State: Initialized, Role: FOLLOWER
I20260928 14:49:08.524869 30105 raft_consensus.cc:348] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.524874 30106 raft_consensus.cc:348] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.525008 30104 consensus_queue.cc:260] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.525156 30105 raft_consensus.cc:374] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:08.525172 30106 raft_consensus.cc:374] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:08.525216 30105 raft_consensus.cc:729] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b754c626274b11b521d4a74b28bdd9, State: Initialized, Role: FOLLOWER
I20260928 14:49:08.525224 30106 raft_consensus.cc:729] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a837633abfca4451b3e3159169b08f7b, State: Initialized, Role: FOLLOWER
I20260928 14:49:08.525758 30106 consensus_queue.cc:260] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.525862 30087 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
I20260928 14:49:08.526515 29953 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
I20260928 14:49:08.526714 30105 consensus_queue.cc:260] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.526912 30106 ts_tablet_manager.cc:1428] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260928 14:49:08.527653 29819 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
I20260928 14:49:08.527984 30105 ts_tablet_manager.cc:1428] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260928 14:49:08.528533 30104 ts_tablet_manager.cc:1428] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20260928 14:49:08.592125 30112 raft_consensus.cc:482] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:08.592410 30112 raft_consensus.cc:504] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.593891 30112 leader_election.cc:288] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647), a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543)
I20260928 14:49:08.598173 30114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.594282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:41647 (local address 127.13.223.65:36339)
0928 14:49:08.594685 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.594696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.594829 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.595108 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.595110 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.595127 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:08.595268 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.595274 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.596069 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.596072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.596970 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.596980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.597690 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.597694 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.597800 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.597831 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.597924 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.598006 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":114,"threads_started":1}
I20260928 14:49:08.598161 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.594330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:36339 (local address 127.13.223.67:41647)
0928 14:49:08.594598 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.594604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.594784 (+ 180us) server_negotiation.cc:407] Connection header received
0928 14:49:08.594948 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.594951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.595003 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.595094 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:08.595389 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.595940 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.597109 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.597547 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.597690 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.597747 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.597941 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.598016 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20260928 14:49:08.598636 30101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.594360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:37503 (local address 127.13.223.66:35543)
0928 14:49:08.594500 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.594505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.594742 (+ 237us) server_negotiation.cc:407] Connection header received
0928 14:49:08.594921 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.594924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.594987 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.595256 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:08.595587 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.596169 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.597445 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.597906 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.598057 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.598126 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.598373 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.598474 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:49:08.599025 30113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.594182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35543 (local address 127.13.223.65:37503)
0928 14:49:08.594620 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.594637 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.594795 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:08.595261 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:08.595264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:08.595283 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:08.595450 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.595458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.596307 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.596312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.597289 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:08.597303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.598048 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.598052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:08.598167 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.598193 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:49:08.598807 (+ 614us) client_negotiation.cc:240] Negotiation successful
0928 14:49:08.598906 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260928 14:49:08.598953 30041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeddb18b0edb454597676f3a3d72be82" candidate_uuid: "64b754c626274b11b521d4a74b28bdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2558fec68df43db95e65d11d52132e4" is_pre_election: true
I20260928 14:49:08.599403 30041 raft_consensus.cc:2454] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b754c626274b11b521d4a74b28bdd9 in term 0.
I20260928 14:49:08.599819 29907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeddb18b0edb454597676f3a3d72be82" candidate_uuid: "64b754c626274b11b521d4a74b28bdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a837633abfca4451b3e3159169b08f7b" is_pre_election: true
I20260928 14:49:08.600028 29707 leader_election.cc:302] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9, e2558fec68df43db95e65d11d52132e4; no voters:
I20260928 14:49:08.600167 29907 raft_consensus.cc:2454] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 64b754c626274b11b521d4a74b28bdd9 in term 0.
I20260928 14:49:08.600395 30112 raft_consensus.cc:2781] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:49:08.600488 30112 raft_consensus.cc:482] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:49:08.600556 30112 raft_consensus.cc:3037] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:08.602183 30112 raft_consensus.cc:504] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.602689 30112 leader_election.cc:288] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 1 election: Requested vote from peers e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647), a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543)
I20260928 14:49:08.603632 30041 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeddb18b0edb454597676f3a3d72be82" candidate_uuid: "64b754c626274b11b521d4a74b28bdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2558fec68df43db95e65d11d52132e4"
I20260928 14:49:08.603641 29907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aeddb18b0edb454597676f3a3d72be82" candidate_uuid: "64b754c626274b11b521d4a74b28bdd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a837633abfca4451b3e3159169b08f7b"
I20260928 14:49:08.603788 30041 raft_consensus.cc:3037] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:08.603814 29907 raft_consensus.cc:3037] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:08.605078 29907 raft_consensus.cc:2454] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b754c626274b11b521d4a74b28bdd9 in term 1.
I20260928 14:49:08.605161 30041 raft_consensus.cc:2454] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 64b754c626274b11b521d4a74b28bdd9 in term 1.
I20260928 14:49:08.605572 29707 leader_election.cc:302] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9, e2558fec68df43db95e65d11d52132e4; no voters:
I20260928 14:49:08.605822 30112 raft_consensus.cc:2781] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:49:08.606128 30112 raft_consensus.cc:686] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 LEADER]: Becoming Leader. State: Replica: 64b754c626274b11b521d4a74b28bdd9, State: Running, Role: LEADER
I20260928 14:49:08.606588 30112 consensus_queue.cc:237] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:08.610338 29624 catalog_manager.cc:5495] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "64b754c626274b11b521d4a74b28bdd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } health_report { overall_health: UNKNOWN } } }
I20260928 14:49:08.626008 30099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:08.621889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.13.223.65:44013)
0928 14:49:08.622033 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:08.622037 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:08.622051 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:49:08.622138 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:08.622140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:08.622209 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:08.622338 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:49:08.622799 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.623351 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.624301 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:08.624545 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:08.624670 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:08.625053 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:49:08.625545 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:49:08.625692 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:08.625760 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:49:08.625839 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:49:08.633564 29907 raft_consensus.cc:1261] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Refusing update from remote peer 64b754c626274b11b521d4a74b28bdd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:08.633608 30041 raft_consensus.cc:1261] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Refusing update from remote peer 64b754c626274b11b521d4a74b28bdd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:08.635279 30115 consensus_queue.cc:1035] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:08.635668 30112 consensus_queue.cc:1035] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:08.644587 30120 mvcc.cc:205] Tried to move back new op lower bound from 7334326061591891968 to 7334326061496037376. Current Snapshot: MvccSnapshot[applied={T|T < 7334326061591891968}]
I20260928 14:49:08.654892 29624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:08.655153 29624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38296:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:08.658406 29624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Sending DeleteTablet for 3 replicas of tablet aeddb18b0edb454597676f3a3d72be82
I20260928 14:49:08.659461 29887 tablet_service.cc:1514] Processing DeleteTablet for tablet aeddb18b0edb454597676f3a3d72be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:33464
I20260928 14:49:08.659623 30021 tablet_service.cc:1514] Processing DeleteTablet for tablet aeddb18b0edb454597676f3a3d72be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:60748
I20260928 14:49:08.660043 30127 tablet_replica.cc:330] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: stopping tablet replica
I20260928 14:49:08.660101 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29690
I20260928 14:49:08.660534 30127 raft_consensus.cc:2229] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:49:08.660954 30128 tablet_replica.cc:330] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: stopping tablet replica
I20260928 14:49:08.661127 30127 raft_consensus.cc:2258] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:08.661212 30128 raft_consensus.cc:2229] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:49:08.661446 30128 raft_consensus.cc:2258] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:08.663719 30127 ts_tablet_manager.cc:1905] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:08.663719 30128 ts_tablet_manager.cc:1905] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:08.667397 30128 ts_tablet_manager.cc:1918] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:08.667395 30127 ts_tablet_manager.cc:1918] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:08.667553 30128 log.cc:1196] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/aeddb18b0edb454597676f3a3d72be82
I20260928 14:49:08.667553 30127 log.cc:1196] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/aeddb18b0edb454597676f3a3d72be82
I20260928 14:49:08.667958 30128 ts_tablet_manager.cc:1939] T aeddb18b0edb454597676f3a3d72be82 P e2558fec68df43db95e65d11d52132e4: Deleting consensus metadata
I20260928 14:49:08.667958 30127 ts_tablet_manager.cc:1939] T aeddb18b0edb454597676f3a3d72be82 P a837633abfca4451b3e3159169b08f7b: Deleting consensus metadata
W20260928 14:49:08.669343 29613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:44013 (error 108)
I20260928 14:49:08.669586 29614 catalog_manager.cc:4841] TS a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): tablet aeddb18b0edb454597676f3a3d72be82 (table test-table [id=23b63db092b3487c8d96193e468599d3]) successfully deleted
I20260928 14:49:08.669924 29613 catalog_manager.cc:4841] TS e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): tablet aeddb18b0edb454597676f3a3d72be82 (table test-table [id=23b63db092b3487c8d96193e468599d3]) successfully deleted
W20260928 14:49:08.670637 30103 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:49:08.670243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44013 (local address 127.0.0.1:59724)
0928 14:49:08.670380 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:08.670399 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:08.670481 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44013: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:49:08.670811 29613 catalog_manager.cc:4584] TS 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet aeddb18b0edb454597676f3a3d72be82: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44013: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 14:49:08.670861 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29824
I20260928 14:49:08.677650 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29958
I20260928 14:49:08.683794 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 29596
I20260928 14:49:28.691339 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--webserver_interface=127.13.223.126
--webserver_port=44253
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:28.808324 30130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:28.808671 30130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:28.808753 30130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:28.813054 30130 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:49:28.813210 30130 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:49:28.813239 30130 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:49:28.813263 30130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:49:28.813305 30130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:28.813354 30130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:28.813410 30130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:49:28.813448 30130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:49:28.819386 30130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:28.820604 30130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:28.821744 30130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:28.836771 30137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:28.836829 30136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:28.836830 30130 server_base.cc:1034] running on GCE node
W20260928 14:49:28.836917 30139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:28.837761 30130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:28.839244 30130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:28.840430 30130 hybrid_clock.cc:648] HybridClock initialized: now 1790606968840407 us; error 42 us; skew 500 ppm
Sep 28 14:49:28 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:28.849800 30130 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:49:28.851043 30130 webserver.cc:460] Webserver started at http://127.13.223.126:44253/ using document root <none> and password file <none>
I20260928 14:49:28.851735 30130 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:28.851820 30130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:28.856443 30130 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:49:28.859665 30146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:28.861330 30130 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:49:28.861541 30130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "a2e6a68d140e4cdb8870c95f6fed42f5"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "4a9d277bcea8223da96b0568d25a810b"
server_key_iv: "00667cb260effcb0dc2b326ad41b633a"
server_key_version: "encryptionkey@0"
I20260928 14:49:28.862247 30130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:28.896853 30130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:28.900252 30130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:28.900480 30130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:28.908603 30130 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44297
I20260928 14:49:28.909034 30198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44297 every 8 connection(s)
I20260928 14:49:28.909889 30130 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:49:28.914832 30199 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:49:28.916718 30199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap starting.
I20260928 14:49:28.920140 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30130
I20260928 14:49:28.920766 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:44013
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=42055
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:49:28.935792 30199 log.cc:824] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:28.945235 30199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:49:28.945676 30199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap complete.
I20260928 14:49:28.950028 30199 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:28.950495 30199 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Initialized, Role: FOLLOWER
I20260928 14:49:28.951155 30199 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:28.951287 30199 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:49:28.951381 30199 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:49:28.951504 30199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 1 FOLLOWER]: Advancing to term 2
I20260928 14:49:28.953773 30199 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:28.954161 30199 leader_election.cc:302] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e6a68d140e4cdb8870c95f6fed42f5; no voters:
I20260928 14:49:28.954515 30199 leader_election.cc:288] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 14:49:28.954648 30203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 14:49:28.954897 30203 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 LEADER]: Becoming Leader. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Running, Role: LEADER
I20260928 14:49:28.955235 30203 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:28.955920 30199 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:49:28.956967 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e6a68d140e4cdb8870c95f6fed42f5. Latest consensus state: current_term: 2 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:28.957161 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:28.957710 30207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:49:28.958110 30204 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:28.958235 30204 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:28.963990 30207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:49:28.964907 30207 catalog_manager.cc:1259] Loaded cluster ID: bf063a928547454fb23526aea2eb0ba0
I20260928 14:49:28.964993 30207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:49:28.968125 30207 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:49:28.969094 30207 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Loaded TSK: 0
I20260928 14:49:28.988840 30207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Generated new TSK 1
I20260928 14:49:28.989194 30207 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:49:29.052759 30201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:29.053097 30201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:29.053208 30201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:29.057826 30201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:29.057924 30201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:29.058022 30201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:49:29.064486 30201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=42055
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:29.065701 30201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:29.066923 30201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:29.077770 30227 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:29.077800 30229 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:29.077770 30226 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:29.078274 30201 server_base.cc:1034] running on GCE node
I20260928 14:49:29.078760 30201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:29.079523 30201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:29.080770 30201 hybrid_clock.cc:648] HybridClock initialized: now 1790606969080742 us; error 45 us; skew 500 ppm
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:29.083966 30201 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:49:29.085109 30201 webserver.cc:460] Webserver started at http://127.13.223.65:42055/ using document root <none> and password file <none>
I20260928 14:49:29.085800 30201 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:29.085947 30201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:29.090678 30201 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:29.093634 30236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.095285 30201 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:29.095450 30201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "64b754c626274b11b521d4a74b28bdd9"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "cd3da4533281b6dbd65ccf3f735e6786"
server_key_iv: "c11b8efe249132a60001427bf2c38ba8"
server_key_version: "encryptionkey@0"
I20260928 14:49:29.096071 30201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:29.118214 30201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:29.122027 30201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:29.122299 30201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:29.123144 30201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:29.124433 30243 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:49:29.129172 30201 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:49:29.129297 30201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260928 14:49:29.129392 30201 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:49:29.131422 30201 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:49:29.131510 30201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 14:49:29.131834 30243 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap starting.
I20260928 14:49:29.143684 30201 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:44013
I20260928 14:49:29.143795 30350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:44013 every 8 connection(s)
I20260928 14:49:29.144816 30201 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:49:29.149210 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30201
I20260928 14:49:29.149880 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:35543
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=33797
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:49:29.152339 30243 log.cc:824] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:29.163887 30243 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:49:29.164379 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.147146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:47837 (local address 127.13.223.126:44297)
0928 14:49:29.147617 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.147641 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.147994 (+ 353us) server_negotiation.cc:407] Connection header received
0928 14:49:29.149263 (+ 1269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.149274 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.149693 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.150086 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.151793 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.153375 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.154738 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.155233 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.155432 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.158372 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.158382 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.158401 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.158445 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.160630 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.161259 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.161266 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.161272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.161423 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.161861 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.161865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.161867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.162275 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.162443 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.163111 (+ 668us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.163386 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":177,"threads_started":1}
I20260928 14:49:29.164403 30354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.147056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:47837)
0928 14:49:29.147776 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.147837 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.148788 (+ 951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.150374 (+ 1586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.150384 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.150885 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:29.151553 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.151582 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.153593 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.153597 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.154500 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.154512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.155406 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.155411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.155636 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.156326 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:29.156346 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:29.158075 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:29.160796 (+ 2721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.160802 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.160816 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.161114 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.161550 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.161554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.161556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.161723 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.162414 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:29.162421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:29.162771 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.162986 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.163379 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":192,"threads_started":1}
I20260928 14:49:29.164403 30243 tablet_bootstrap.cc:492] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap complete.
I20260928 14:49:29.166800 30243 ts_tablet_manager.cc:1397] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Time spent bootstrapping tablet: real 0.035s user 0.018s sys 0.015s
I20260928 14:49:29.168363 30351 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:29.168671 30351 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:29.169287 30351 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
I20260928 14:49:29.169967 30243 raft_consensus.cc:348] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:29.170454 30243 raft_consensus.cc:729] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b754c626274b11b521d4a74b28bdd9, State: Initialized, Role: FOLLOWER
I20260928 14:49:29.171066 30243 consensus_queue.cc:260] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } }
I20260928 14:49:29.172160 30243 ts_tablet_manager.cc:1428] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260928 14:49:29.172663 30163 ts_manager.cc:194] Registered new tserver with Master: 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013)
I20260928 14:49:29.177302 30163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:47837
I20260928 14:49:29.180857 30351 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606968, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.65@KRBTEST.COM
I20260928 14:49:29.188288 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.176645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44013 (local address 127.0.0.1:49768)
0928 14:49:29.177320 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.177378 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.177527 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.178064 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.178070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.178607 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:29.178736 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.178743 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.179925 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.179928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.180583 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.180593 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.181335 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.181339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.181468 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.182149 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:29.182163 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:29.183926 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:29.186091 (+ 2165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.186103 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.186110 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.186399 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.186836 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.186840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.186842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.186945 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.187498 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:29.187505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:29.187657 (+ 152us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.187820 (+ 163us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.188066 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":115,"threads_started":1}
I20260928 14:49:29.188288 30359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.176755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.13.223.65:44013)
0928 14:49:29.177603 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.177618 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.177645 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:49:29.177728 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.177732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.177904 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.178068 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.178866 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.179802 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.180730 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.181179 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.181322 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.184120 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.184129 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.184142 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.184171 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.185947 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.186604 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.186611 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.186613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.186723 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.187059 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.187063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.187065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.187365 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.187527 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.187914 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.188108 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":651,"thread_start_us":108,"threads_started":1}
I20260928 14:49:29.189518 30285 tablet_service.cc:1514] Processing DeleteTablet for tablet aeddb18b0edb454597676f3a3d72be82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:49768
I20260928 14:49:29.190754 30360 tablet_replica.cc:330] stopping tablet replica
I20260928 14:49:29.191088 30360 raft_consensus.cc:2229] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:49:29.191354 30360 raft_consensus.cc:2258] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:29.192350 30360 ts_tablet_manager.cc:1905] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:29.196204 30360 ts_tablet_manager.cc:1918] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:29.196329 30360 log.cc:1196] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal/wals/aeddb18b0edb454597676f3a3d72be82
I20260928 14:49:29.196605 30360 ts_tablet_manager.cc:1939] T aeddb18b0edb454597676f3a3d72be82 P 64b754c626274b11b521d4a74b28bdd9: Deleting consensus metadata
I20260928 14:49:29.198819 30148 catalog_manager.cc:4841] TS 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013): tablet aeddb18b0edb454597676f3a3d72be82 (table test-table [id=23b63db092b3487c8d96193e468599d3]) successfully deleted
W20260928 14:49:29.288379 30355 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:29.288709 30355 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:29.288802 30355 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:29.293120 30355 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:29.293207 30355 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:29.293318 30355 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:49:29.300065 30355 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:35543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=33797
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30355
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:29.301496 30355 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:29.302722 30355 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:29.313823 30366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:29.313823 30368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:29.313872 30355 server_base.cc:1034] running on GCE node
W20260928 14:49:29.313823 30365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:29.314775 30355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:29.315565 30355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:29.316793 30355 hybrid_clock.cc:648] HybridClock initialized: now 1790606969316777 us; error 31 us; skew 500 ppm
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:29.320210 30355 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:49:29.321391 30355 webserver.cc:460] Webserver started at http://127.13.223.66:33797/ using document root <none> and password file <none>
I20260928 14:49:29.322144 30355 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:29.322280 30355 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:29.327255 30355 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:29.329835 30375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.331444 30355 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:29.331612 30355 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "a837633abfca4451b3e3159169b08f7b"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "2562ba80004241680e2a623bfd350172"
server_key_iv: "4e743620ec14604a7429e240320de3ba"
server_key_version: "encryptionkey@0"
I20260928 14:49:29.332275 30355 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:29.349373 30355 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:29.353209 30355 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:29.353430 30355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:29.354105 30355 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:29.355103 30355 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:29.355178 30355 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.355255 30355 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:29.355307 30355 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.365563 30355 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35543
I20260928 14:49:29.365665 30488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35543 every 8 connection(s)
I20260928 14:49:29.366861 30355 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:49:29.369016 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30355
I20260928 14:49:29.369495 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:41647
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=46705
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:29.384444 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.368933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:41451 (local address 127.13.223.126:44297)
0928 14:49:29.369196 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.369202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.370218 (+ 1016us) server_negotiation.cc:407] Connection header received
0928 14:49:29.371847 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.371852 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.371918 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.372030 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.373389 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.373968 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.376027 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.376273 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.376364 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.379941 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.379945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.379947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.380003 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.381658 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.382439 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.382442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.382443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.382529 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.383126 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.383130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.383131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.383349 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.383437 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.384168 (+ 731us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.384313 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:49:29.385507 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.369357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:41451)
0928 14:49:29.369943 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.370022 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.371608 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.372265 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.372273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.372669 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:29.373230 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.373245 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.374133 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.374140 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.375855 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.375872 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.376466 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.376472 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.376751 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.377474 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:29.377504 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:29.379682 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:29.381806 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.381817 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.381838 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.382286 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.382702 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.382707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.382710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.382923 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.383460 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:29.383467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:29.383828 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.384043 (+ 215us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.384426 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":152,"threads_started":1}
I20260928 14:49:29.386922 30489 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:29.387221 30489 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:29.387750 30489 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
I20260928 14:49:29.388903 30163 ts_manager.cc:194] Registered new tserver with Master: a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543)
I20260928 14:49:29.389708 30163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:41451
W20260928 14:49:29.492506 30492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:29.492828 30492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:29.492877 30492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:29.497174 30492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:29.497260 30492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:29.497324 30492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:49:29.503304 30492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:41647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=46705
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:29.504433 30492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:29.505451 30492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:29.515801 30498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:29.515832 30497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:29.515795 30492 server_base.cc:1034] running on GCE node
W20260928 14:49:29.515776 30500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:29.516667 30492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:29.517400 30492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:29.518630 30492 hybrid_clock.cc:648] HybridClock initialized: now 1790606969518585 us; error 61 us; skew 500 ppm
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:29.521522 30492 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:49:29.522511 30492 webserver.cc:460] Webserver started at http://127.13.223.67:46705/ using document root <none> and password file <none>
I20260928 14:49:29.523180 30492 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:29.523301 30492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:29.527376 30492 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 14:49:29.529745 30507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.531227 30492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:29.531385 30492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "e2558fec68df43db95e65d11d52132e4"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "dd46f7e90beaa60be05ecb4943715ebf"
server_key_iv: "e26f0e5ccaf2c26b62a6f92256862a63"
server_key_version: "encryptionkey@0"
I20260928 14:49:29.532011 30492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:29.560619 30492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:29.564328 30492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:29.564563 30492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:29.565272 30492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:29.566219 30492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:29.566293 30492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.566371 30492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:29.566408 30492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:29.576603 30492 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:41647
I20260928 14:49:29.576627 30620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:41647 every 8 connection(s)
I20260928 14:49:29.577692 30492 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:49:29.578984 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30492
Sep 28 14:49:29 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606969, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:49:29.588874 30353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.579758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44998 (local address 127.13.223.126:44297)
0928 14:49:29.579912 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.579917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.580181 (+ 264us) server_negotiation.cc:407] Connection header received
0928 14:49:29.580283 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.580286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.580336 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.580431 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.581377 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.581930 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.583251 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.583498 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.583594 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.585420 (+ 1826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.585423 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.585425 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.585449 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.587145 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.587633 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.587638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.587640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.587723 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.588014 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.588019 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.588020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.588200 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.588427 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.588575 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.588697 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:49:29.593856 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.579840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:56689 (local address 127.13.223.126:44297)
0928 14:49:29.580029 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.580033 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.581135 (+ 1102us) server_negotiation.cc:407] Connection header received
0928 14:49:29.582685 (+ 1550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.582688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.582736 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.582815 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.584401 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.585058 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.586318 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.586679 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.586764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.589707 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.589710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.589712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.589732 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.591420 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.592071 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.592073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.592075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.592188 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.592535 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.592538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.592539 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.592745 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.592826 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.593565 (+ 739us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.593696 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":57,"threads_started":1}
I20260928 14:49:29.594866 30625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.580336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:56689)
0928 14:49:29.580944 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.581007 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.582417 (+ 1410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.583049 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.583058 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.583520 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:29.584228 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.584245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.585220 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.585224 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.586217 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.586228 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.586842 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.586846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.587094 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.587773 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:29.587790 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:29.589543 (+ 1753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:29.591563 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.591569 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.591584 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.591904 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.592280 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:29.592282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:29.592284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:29.592444 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:29.592923 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:29.592932 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:29.593261 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.593443 (+ 182us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.593757 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":145,"threads_started":1}
I20260928 14:49:29.596403 30621 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44297
I20260928 14:49:29.596669 30621 heartbeater.cc:464] Registering TS with master...
I20260928 14:49:29.597195 30621 heartbeater.cc:511] Master 127.13.223.126:44297 requested a full tablet report, sending...
I20260928 14:49:29.598331 30163 ts_manager.cc:194] Registered new tserver with Master: e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
I20260928 14:49:29.599022 30163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:56689
I20260928 14:49:29.603312 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:49:29.608436 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.604047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45012 (local address 127.13.223.126:44297)
0928 14:49:29.604232 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.604237 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.604479 (+ 242us) server_negotiation.cc:407] Connection header received
0928 14:49:29.604747 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.604750 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.604865 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.604996 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:49:29.605293 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.605888 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.606995 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.607265 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.607395 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.607776 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:49:29.607937 (+ 161us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 14:49:29.608264 (+ 327us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45012: token expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 14:49:29.608547 30624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45012: token expired
W20260928 14:49:29.609117 30092 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.13.223.126:44297)
I20260928 14:49:29.618006 30624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.609718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.13.223.126:44297)
0928 14:49:29.609860 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.609865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.609881 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:49:29.609932 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.609936 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.609989 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.610087 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:29.611033 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.611661 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.612714 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.612971 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.613133 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.614705 (+ 1572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:29.614709 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:29.614711 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:29.614740 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:29.616342 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.616796 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.616799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.616800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.616895 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:29.617141 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:29.617143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:29.617144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:29.617335 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:29.617440 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.617657 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.617840 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:49:29.620800 30163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:49:29.624297 30163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:49:29.633169 30285 tablet_service.cc:1467] Processing CreateTablet for tablet 65ce9ba82a6542f39e9e27696d075820 (DEFAULT_TABLE table=test-table [id=353acdb17d3e4d8183358ec0810f48fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:29.633837 30285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ce9ba82a6542f39e9e27696d075820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:29.637063 30243 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap starting.
I20260928 14:49:29.637516 30358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.631644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35543 (local address 127.0.0.1:51846)
0928 14:49:29.631781 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.631798 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.631887 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.632751 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.632754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.632773 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:29.632891 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.632898 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.634651 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.634657 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.635669 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.635702 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.637035 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.637038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.637154 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.637220 (+ 66us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.637274 (+ 54us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.637364 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 14:49:29.637854 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.632140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:41647 (local address 127.0.0.1:53012)
0928 14:49:29.632367 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.632380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.632468 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.633204 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.633210 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.633229 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:29.633315 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.633321 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.634688 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.634690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.635980 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.635994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.637298 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.637301 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.637384 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.637402 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.637446 (+ 44us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.637497 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":156,"thread_start_us":99,"threads_started":1}
I20260928 14:49:29.638038 30243 tablet_bootstrap.cc:654] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:29.638100 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.632296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.13.223.67:41647)
0928 14:49:29.632740 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.632753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.632775 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:49:29.632858 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.632863 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.633047 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.633208 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:29.633411 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.634536 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.636109 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.637163 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.637592 (+ 429us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.637776 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.637830 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.637933 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":167,"threads_started":1}
I20260928 14:49:29.638615 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.631832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51846 (local address 127.13.223.66:35543)
0928 14:49:29.632201 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.632214 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.632241 (+ 27us) server_negotiation.cc:407] Connection header received
0928 14:49:29.632333 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.632339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.632603 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.632777 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:29.633224 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.634477 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.635901 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.636893 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.638108 (+ 1215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.638288 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.638344 (+ 56us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.638439 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":179,"threads_started":1}
I20260928 14:49:29.639925 30243 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: No bootstrap required, opened a new log
I20260928 14:49:29.640002 30243 ts_tablet_manager.cc:1397] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:29.640424 30243 raft_consensus.cc:348] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.640530 30243 raft_consensus.cc:374] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:29.640558 30243 raft_consensus.cc:729] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b754c626274b11b521d4a74b28bdd9, State: Initialized, Role: FOLLOWER
I20260928 14:49:29.640771 30243 consensus_queue.cc:260] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.641096 30243 ts_tablet_manager.cc:1428] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 14:49:29.641320 30555 tablet_service.cc:1467] Processing CreateTablet for tablet 65ce9ba82a6542f39e9e27696d075820 (DEFAULT_TABLE table=test-table [id=353acdb17d3e4d8183358ec0810f48fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:29.642589 30555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ce9ba82a6542f39e9e27696d075820. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:49:29.644889 30423 tablet_service.cc:1467] Processing CreateTablet for tablet 65ce9ba82a6542f39e9e27696d075820 (DEFAULT_TABLE table=test-table [id=353acdb17d3e4d8183358ec0810f48fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:49:29.645891 30423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65ce9ba82a6542f39e9e27696d075820. 1 dirs total, 0 dirs full, 0 dirs failed
W20260928 14:49:29.646162 30352 tablet.cc:2369] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:49:29.650233 30634 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Bootstrap starting.
I20260928 14:49:29.653535 30634 tablet_bootstrap.cc:654] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:29.653913 30635 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Bootstrap starting.
I20260928 14:49:29.654632 30634 log.cc:824] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:29.656741 30635 tablet_bootstrap.cc:654] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Neither blocks nor log segments found. Creating new log.
I20260928 14:49:29.657025 30634 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: No bootstrap required, opened a new log
I20260928 14:49:29.657259 30634 ts_tablet_manager.cc:1397] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:49:29.657675 30635 log.cc:824] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:29.659876 30635 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: No bootstrap required, opened a new log
I20260928 14:49:29.660147 30635 ts_tablet_manager.cc:1397] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:49:29.660718 30634 raft_consensus.cc:348] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.661007 30634 raft_consensus.cc:374] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:29.661096 30634 raft_consensus.cc:729] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2558fec68df43db95e65d11d52132e4, State: Initialized, Role: FOLLOWER
I20260928 14:49:29.661648 30634 consensus_queue.cc:260] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.662400 30621 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
I20260928 14:49:29.662828 30634 ts_tablet_manager.cc:1428] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:49:29.663890 30635 raft_consensus.cc:348] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.664186 30635 raft_consensus.cc:374] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:49:29.664247 30635 raft_consensus.cc:729] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a837633abfca4451b3e3159169b08f7b, State: Initialized, Role: FOLLOWER
I20260928 14:49:29.664829 30635 consensus_queue.cc:260] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.665532 30489 heartbeater.cc:503] Master 127.13.223.126:44297 was elected leader, sending a full tablet report...
I20260928 14:49:29.665970 30635 ts_tablet_manager.cc:1428] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 14:49:29.723186 30639 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:29.723438 30639 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.724759 30639 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647), 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013)
I20260928 14:49:29.729231 30491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.725095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:41647 (local address 127.13.223.66:39793)
0928 14:49:29.725287 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.725302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.725442 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.725803 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.725807 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.725831 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:29.726000 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.726008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.727027 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.727031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.727954 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.727964 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.728744 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.728869 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.728904 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.728978 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.729061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":1}
I20260928 14:49:29.729314 30632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.725287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:39793 (local address 127.13.223.67:41647)
0928 14:49:29.725442 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.725448 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.725464 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:49:29.725534 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.725539 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.725590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.725674 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:29.726201 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.726910 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728101 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.728621 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728722 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.728765 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.729016 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.729100 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 14:49:29.729377 30641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.725559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:36823 (local address 127.13.223.65:44013)
0928 14:49:29.725861 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.725865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.725880 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:49:29.726003 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.726006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.726085 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.726196 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:49:29.726499 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.727185 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728296 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.728815 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728945 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.729121 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.729164 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.729237 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225,"thread_start_us":77,"threads_started":1}
I20260928 14:49:29.729442 30640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.725412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44013 (local address 127.13.223.66:36823)
0928 14:49:29.725815 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.725827 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:29.725897 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:29.726230 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:29.726233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:29.726253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:49:29.726369 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.726375 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.727302 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.727304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.728168 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:29.728178 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.728934 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.728936 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:29.729070 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.729091 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 14:49:29.729156 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 14:49:29.729222 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":1,"thread_start_us":134,"threads_started":1}
I20260928 14:49:29.729966 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ce9ba82a6542f39e9e27696d075820" candidate_uuid: "a837633abfca4451b3e3159169b08f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b754c626274b11b521d4a74b28bdd9" is_pre_election: true
I20260928 14:49:29.730010 30575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ce9ba82a6542f39e9e27696d075820" candidate_uuid: "a837633abfca4451b3e3159169b08f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2558fec68df43db95e65d11d52132e4" is_pre_election: true
I20260928 14:49:29.730296 30305 raft_consensus.cc:2454] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a837633abfca4451b3e3159169b08f7b in term 0.
I20260928 14:49:29.730310 30575 raft_consensus.cc:2454] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a837633abfca4451b3e3159169b08f7b in term 0.
I20260928 14:49:29.730821 30377 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9, a837633abfca4451b3e3159169b08f7b; no voters:
I20260928 14:49:29.731158 30639 raft_consensus.cc:2781] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:49:29.731241 30639 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:49:29.731272 30639 raft_consensus.cc:3037] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:29.732416 30639 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.732829 30639 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [CANDIDATE]: Term 1 election: Requested vote from peers e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647), 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013)
I20260928 14:49:29.733511 30575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ce9ba82a6542f39e9e27696d075820" candidate_uuid: "a837633abfca4451b3e3159169b08f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2558fec68df43db95e65d11d52132e4"
I20260928 14:49:29.733531 30305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65ce9ba82a6542f39e9e27696d075820" candidate_uuid: "a837633abfca4451b3e3159169b08f7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64b754c626274b11b521d4a74b28bdd9"
I20260928 14:49:29.733681 30575 raft_consensus.cc:3037] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:29.733690 30305 raft_consensus.cc:3037] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:49:29.734933 30305 raft_consensus.cc:2454] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a837633abfca4451b3e3159169b08f7b in term 1.
I20260928 14:49:29.735013 30575 raft_consensus.cc:2454] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a837633abfca4451b3e3159169b08f7b in term 1.
I20260928 14:49:29.735301 30377 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9, a837633abfca4451b3e3159169b08f7b; no voters:
I20260928 14:49:29.735522 30639 raft_consensus.cc:2781] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:49:29.735850 30639 raft_consensus.cc:686] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 1 LEADER]: Becoming Leader. State: Replica: a837633abfca4451b3e3159169b08f7b, State: Running, Role: LEADER
I20260928 14:49:29.736322 30639 consensus_queue.cc:237] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:29.740281 30162 catalog_manager.cc:5495] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b reported cstate change: term changed from 0 to 1, leader changed from <none> to a837633abfca4451b3e3159169b08f7b (127.13.223.66). New cstate: current_term: 1 leader_uuid: "a837633abfca4451b3e3159169b08f7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } health_report { overall_health: UNKNOWN } } }
I20260928 14:49:29.775563 30630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.770456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51856 (local address 127.13.223.66:35543)
0928 14:49:29.770715 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:29.770720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:29.770959 (+ 239us) server_negotiation.cc:407] Connection header received
0928 14:49:29.771174 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:29.771179 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:29.771269 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:29.771406 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:49:29.771765 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.772549 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.773519 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:29.773878 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:29.773989 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:29.774393 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:49:29.775041 (+ 648us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:49:29.775182 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:29.775259 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 14:49:29.775346 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260928 14:49:29.783282 30575 raft_consensus.cc:1261] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Refusing update from remote peer a837633abfca4451b3e3159169b08f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:29.783403 30305 raft_consensus.cc:1261] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Refusing update from remote peer a837633abfca4451b3e3159169b08f7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:49:29.784250 30642 consensus_queue.cc:1035] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:29.784487 30639 consensus_queue.cc:1035] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:49:29.796017 30647 mvcc.cc:205] Tried to move back new op lower bound from 7334326148222574592 to 7334326148043595776. Current Snapshot: MvccSnapshot[applied={T|T < 7334326148222574592}]
I20260928 14:49:29.801384 30163 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:29.801592 30163 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:49:29.804378 30163 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Sending DeleteTablet for 3 replicas of tablet 65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:29.805395 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30201
I20260928 14:49:29.807578 30423 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ce9ba82a6542f39e9e27696d075820 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:29 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51846
I20260928 14:49:29.808211 30655 tablet_replica.cc:330] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: stopping tablet replica
I20260928 14:49:29.808276 30555 tablet_service.cc:1514] Processing DeleteTablet for tablet 65ce9ba82a6542f39e9e27696d075820 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:49:29 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:53012
I20260928 14:49:29.808825 30656 tablet_replica.cc:330] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: stopping tablet replica
I20260928 14:49:29.808836 30655 raft_consensus.cc:2229] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:49:29.809144 30656 raft_consensus.cc:2229] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:49:29.809412 30655 raft_consensus.cc:2258] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:29.809513 30656 raft_consensus.cc:2258] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:49:29.811237 30656 ts_tablet_manager.cc:1905] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:29.811615 30655 ts_tablet_manager.cc:1905] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:49:29.814039 30148 connection.cc:535] client connection to 127.13.223.65:44013 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:49:29.814201 30656 ts_tablet_manager.cc:1918] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260928 14:49:29.814229 30148 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:49:29.814317 30656 log.cc:1196] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:29.814740 30656 ts_tablet_manager.cc:1939] T 65ce9ba82a6542f39e9e27696d075820 P e2558fec68df43db95e65d11d52132e4: Deleting consensus metadata
I20260928 14:49:29.815361 30655 ts_tablet_manager.cc:1918] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:29.815418 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30355
I20260928 14:49:29.817514 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.817200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:44013 (local address 127.0.0.1:49780)
0928 14:49:29.817351 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.817428 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44013: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 14:49:29.817767 30148 catalog_manager.cc:4584] TS 64b754c626274b11b521d4a74b28bdd9 (127.13.223.65:44013): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 65ce9ba82a6542f39e9e27696d075820: Network error: Client connection negotiation failed: client connection to 127.13.223.65:44013: connect: Connection refused (error 111)
I20260928 14:49:29.818686 30148 catalog_manager.cc:4841] TS e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): tablet 65ce9ba82a6542f39e9e27696d075820 (table test-table [id=353acdb17d3e4d8183358ec0810f48fa]) successfully deleted
I20260928 14:49:29.823868 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30492
I20260928 14:49:29.826143 30631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:29.824168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35543 (local address 127.0.0.1:51858)
0928 14:49:29.825960 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:29.826036 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:35543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1663}
I20260928 14:49:29.830268 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30130
I20260928 14:49:49.839620 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--webserver_interface=127.13.223.126
--webserver_port=44253
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44297
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:49:49.998595 30658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:49.999034 30658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:49.999084 30658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:50.005708 30658 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 14:49:50.005880 30658 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 14:49:50.005906 30658 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 14:49:50.005923 30658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:49:50.005947 30658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:50.006007 30658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:50.006040 30658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:49:50.006168 30658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:49:50.013726 30658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44297
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=44253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:50.015115 30658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:50.016317 30658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:50.031013 30663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.031013 30666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.031076 30664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.031510 30658 server_base.cc:1034] running on GCE node
I20260928 14:49:50.032526 30658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:50.034017 30658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:50.035279 30658 hybrid_clock.cc:648] HybridClock initialized: now 1790606990035242 us; error 55 us; skew 500 ppm
Sep 28 14:49:50 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:50.040074 30658 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:49:50.041780 30658 webserver.cc:460] Webserver started at http://127.13.223.126:44253/ using document root <none> and password file <none>
I20260928 14:49:50.042542 30658 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:50.042745 30658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:50.048549 30658 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:49:50.051661 30673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.053666 30658 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:49:50.053983 30658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "a2e6a68d140e4cdb8870c95f6fed42f5"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "4a9d277bcea8223da96b0568d25a810b"
server_key_iv: "00667cb260effcb0dc2b326ad41b633a"
server_key_version: "encryptionkey@0"
I20260928 14:49:50.054761 30658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:50.075268 30658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:50.080116 30658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:50.080534 30658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:50.091521 30658 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44297
I20260928 14:49:50.092093 30725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44297 every 8 connection(s)
I20260928 14:49:50.092983 30658 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:49:50.098541 30726 sys_catalog.cc:263] Verifying existing consensus state
I20260928 14:49:50.099434 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30658
I20260928 14:49:50.099964 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:44013
--local_ip_for_outbound_sockets=127.13.223.65
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=42055
--webserver_interface=127.13.223.65
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:49:50.100912 30726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap starting.
I20260928 14:49:50.128019 30726 log.cc:824] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:50.141479 30726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:49:50.142012 30726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Bootstrap complete.
I20260928 14:49:50.147513 30726 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:50.148065 30726 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Initialized, Role: FOLLOWER
I20260928 14:49:50.148756 30726 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:50.148905 30726 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:49:50.149024 30726 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:49:50.149183 30726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 2 FOLLOWER]: Advancing to term 3
I20260928 14:49:50.151716 30726 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:50.152189 30726 leader_election.cc:302] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e6a68d140e4cdb8870c95f6fed42f5; no voters:
I20260928 14:49:50.152640 30726 leader_election.cc:288] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 14:49:50.152809 30730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 14:49:50.153088 30730 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [term 3 LEADER]: Becoming Leader. State: Replica: a2e6a68d140e4cdb8870c95f6fed42f5, State: Running, Role: LEADER
I20260928 14:49:50.153450 30730 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } }
I20260928 14:49:50.154166 30726 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:49:50.155191 30731 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:50.155376 30731 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:50.155722 30732 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e6a68d140e4cdb8870c95f6fed42f5. Latest consensus state: current_term: 3 leader_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e6a68d140e4cdb8870c95f6fed42f5" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44297 } } }
I20260928 14:49:50.155898 30732 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5 [sys.catalog]: This master's current role is: LEADER
I20260928 14:49:50.156483 30739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:49:50.163071 30739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:49:50.163914 30739 catalog_manager.cc:1259] Loaded cluster ID: bf063a928547454fb23526aea2eb0ba0
I20260928 14:49:50.164034 30739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:49:50.167764 30739 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:49:50.168793 30739 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Loaded TSK: 0
I20260928 14:49:50.168905 30739 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2e6a68d140e4cdb8870c95f6fed42f5: Loaded TSK: 1
F20260928 14:49:50.169740 30739 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff56ab669ad google::LogMessage::Fail() at ??:0
@ 0x7ff56ab6988a google::LogMessage::SendToLog() at ??:0
@ 0x7ff56ab664de google::LogMessage::Flush() at ??:0
@ 0x7ff56ab67a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff56d82a18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7ff56d822f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7ff5754cf7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7ff5754a999c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7ff5754aa178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7ff5754ebdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7ff57550a82b std::function<>::operator()() at ??:0
@ 0x7ff5754a9b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff5754aaf3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff5754a7579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff5754eb640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff56cdb83d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff56cdb8c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff56cdba3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff56cda9512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff56e3ed6db start_thread at ??:0
@ 0x7ff56966461f clone at ??:0
W20260928 14:49:50.296121 30728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:50.296581 30728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:50.296663 30728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:50.303574 30728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:50.303690 30728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:50.303828 30728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:49:50.310437 30728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:44013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=42055
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:50.312295 30728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:50.313613 30728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:50.325465 30773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.325671 30728 server_base.cc:1034] running on GCE node
W20260928 14:49:50.325418 30774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.325402 30776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.326383 30728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:50.327350 30728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:50.328599 30728 hybrid_clock.cc:648] HybridClock initialized: now 1790606990328565 us; error 59 us; skew 500 ppm
Sep 28 14:49:50 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:50.331972 30728 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:49:50.333101 30728 webserver.cc:460] Webserver started at http://127.13.223.65:42055/ using document root <none> and password file <none>
I20260928 14:49:50.333755 30728 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:50.333858 30728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:50.338724 30728 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:49:50.341530 30783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.343644 30728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:49:50.343845 30728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "64b754c626274b11b521d4a74b28bdd9"
format_stamp: "Formatted at 2026-09-28 14:49:07 on dist-test-slave-gkhx"
server_key: "cd3da4533281b6dbd65ccf3f735e6786"
server_key_iv: "c11b8efe249132a60001427bf2c38ba8"
server_key_version: "encryptionkey@0"
I20260928 14:49:50.344547 30728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:50.364907 30728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:50.368898 30728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:50.369153 30728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:50.369837 30728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:50.371215 30790 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 14:49:50.376168 30728 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 14:49:50.376245 30728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 14:49:50.376303 30728 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 14:49:50.378528 30728 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 14:49:50.378635 30728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260928 14:49:50.378955 30790 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap starting.
I20260928 14:49:50.390635 30728 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:44013
I20260928 14:49:50.390669 30897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:44013 every 8 connection(s)
I20260928 14:49:50.392334 30728 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:49:50.392968 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30728
I20260928 14:49:50.393488 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:35543
--local_ip_for_outbound_sockets=127.13.223.66
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=33797
--webserver_interface=127.13.223.66
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:49:50.404217 30790 log.cc:824] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:49:50.414649 30790 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 14:49:50.415179 30790 tablet_bootstrap.cc:492] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Bootstrap complete.
I20260928 14:49:50.416690 30790 ts_tablet_manager.cc:1397] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Time spent bootstrapping tablet: real 0.038s user 0.031s sys 0.004s
I20260928 14:49:50.420305 30790 raft_consensus.cc:348] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:50.420948 30790 raft_consensus.cc:729] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b754c626274b11b521d4a74b28bdd9, State: Initialized, Role: FOLLOWER
I20260928 14:49:50.421551 30790 consensus_queue.cc:260] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:50.422712 30790 ts_tablet_manager.cc:1428] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260928 14:49:50.440102 30901 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:49:50.396204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:41983)
0928 14:49:50.397307 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.397380 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:50.398537 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:50.439240 (+ 40703us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":823,"thread_start_us":197,"threads_started":1}
W20260928 14:49:50.440822 30898 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44297 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:44297: Client connection negotiation failed: client connection to 127.13.223.126:44297: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 14:49:50.441695 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.441382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:34723)
0928 14:49:50.441539 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.441627 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 14:49:50.442625 30901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.442301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:43441)
0928 14:49:50.442428 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.442489 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:49:50.442829 30898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:49:50.531838 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:50.532349 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:50.532459 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:50.537830 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:50.537993 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:50.538117 30900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:49:50.544866 30900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:35543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=33797
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:50.546193 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:50.547470 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:50.559494 30908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.559510 30909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.559691 30900 server_base.cc:1034] running on GCE node
W20260928 14:49:50.559494 30911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.560411 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:50.561300 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:50.562541 30900 hybrid_clock.cc:648] HybridClock initialized: now 1790606990562497 us; error 77 us; skew 500 ppm
Sep 28 14:49:50 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:50.566619 30900 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:49:50.568092 30900 webserver.cc:460] Webserver started at http://127.13.223.66:33797/ using document root <none> and password file <none>
I20260928 14:49:50.568902 30900 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:50.569041 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:50.574101 30900 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:49:50.576939 30918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.578912 30900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:50.579131 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "a837633abfca4451b3e3159169b08f7b"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "2562ba80004241680e2a623bfd350172"
server_key_iv: "4e743620ec14604a7429e240320de3ba"
server_key_version: "encryptionkey@0"
I20260928 14:49:50.579864 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:50.602309 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:50.606714 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:50.607017 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:50.607718 30900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:50.609091 30925 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260928 14:49:50.613411 30925 ts_tablet_manager.cc:1852] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260928 14:49:50.613848 30925 ts_tablet_manager.cc:1905] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:49:50.618943 30925 ts_tablet_manager.cc:1918] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:49:50.619180 30925 log.cc:1196] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:50.619647 30925 ts_tablet_manager.cc:1939] T 65ce9ba82a6542f39e9e27696d075820 P a837633abfca4451b3e3159169b08f7b: Deleting consensus metadata
I20260928 14:49:50.620281 30900 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260928 14:49:50.620388 30900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.000s
I20260928 14:49:50.620517 30900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:50.620570 30900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.631590 30900 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35543
I20260928 14:49:50.631688 31032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35543 every 8 connection(s)
I20260928 14:49:50.632648 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:49:50.634328 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 30900
I20260928 14:49:50.635025 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:41647
--local_ip_for_outbound_sockets=127.13.223.67
--tserver_master_addrs=127.13.223.126:44297
--webserver_port=46705
--webserver_interface=127.13.223.67
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:49:50.637802 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.635727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:55961)
0928 14:49:50.636447 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.636617 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":254,"threads_started":1}
W20260928 14:49:50.638368 31033 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44297 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:44297: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
I20260928 14:49:50.639170 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.638855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:44599)
0928 14:49:50.638990 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.639109 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:49:50.640168 31035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.639801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:42047)
0928 14:49:50.640046 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.640104 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260928 14:49:50.640398 31033 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 14:49:50.759785 31036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:49:50.760111 31036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:49:50.760164 31036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:49:50.764706 31036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:49:50.764791 31036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:49:50.764856 31036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:49:50.770918 31036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:41647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=46705
--tserver_master_addrs=127.13.223.126:44297
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:49:50.772389 31036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:49:50.773701 31036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:49:50.787827 31041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.788249 31042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:49:50.788409 31044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:49:50.790692 31036 server_base.cc:1034] running on GCE node
I20260928 14:49:50.791182 31036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:49:50.791864 31036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:49:50.793121 31036 hybrid_clock.cc:648] HybridClock initialized: now 1790606990793099 us; error 36 us; skew 500 ppm
Sep 28 14:49:50 dist-test-slave-gkhx krb5kdc[29583](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:49:50.796933 31036 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:49:50.798061 31036 webserver.cc:460] Webserver started at http://127.13.223.67:46705/ using document root <none> and password file <none>
I20260928 14:49:50.798792 31036 fs_manager.cc:359] Metadata directory not provided
I20260928 14:49:50.798899 31036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:49:50.803443 31036 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:49:50.806908 31051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.808971 31036 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:49:50.809231 31036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "e2558fec68df43db95e65d11d52132e4"
format_stamp: "Formatted at 2026-09-28 14:49:08 on dist-test-slave-gkhx"
server_key: "dd46f7e90beaa60be05ecb4943715ebf"
server_key_iv: "e26f0e5ccaf2c26b62a6f92256862a63"
server_key_version: "encryptionkey@0"
I20260928 14:49:50.809897 31036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:49:50.834540 31036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:49:50.838249 31036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:49:50.838505 31036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:49:50.839259 31036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:49:50.840217 31036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:49:50.840298 31036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.840381 31036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:49:50.840426 31036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:49:50.851939 31036 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:41647
I20260928 14:49:50.851959 31164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:41647 every 8 connection(s)
I20260928 14:49:50.853070 31036 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:49:50.857398 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.855466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:41847)
0928 14:49:50.855966 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.856151 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":106,"threads_started":1}
I20260928 14:49:50.858027 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31036
W20260928 14:49:50.858101 31165 heartbeater.cc:644] Failed to heartbeat to 127.13.223.126:44297 (0 consecutive failures): Network error: Failed to ping master at 127.13.223.126:44297: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
I20260928 14:49:50.859115 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.858738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:48105)
0928 14:49:50.858964 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.859031 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260928 14:49:50.859922 31167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:50.859722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:45961)
0928 14:49:50.859827 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:50.859870 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:49:50.860212 31165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 14:49:51.444175 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.443655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:39417)
0928 14:49:51.443983 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:51.444051 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":85,"threads_started":1}
I20260928 14:49:51.641716 31170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.641249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:49507)
0928 14:49:51.641560 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:51.641620 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":82,"threads_started":1}
I20260928 14:49:51.861331 31171 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:51.861611 31172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.861051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:40325)
0928 14:49:51.861343 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:51.861426 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":117,"threads_started":1}
I20260928 14:49:51.861614 31171 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:51.863054 31171 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543), e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
Sep 28 14:49:51 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.66@KRBTEST.COM
Sep 28 14:49:51 dist-test-slave-gkhx krb5kdc[29583](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790606990, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.67@KRBTEST.COM
I20260928 14:49:51.889269 31174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.863423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:55587 (local address 127.13.223.66:35543)
0928 14:49:51.863938 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:51.863956 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:51.864007 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:49:51.865054 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:51.865065 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:51.865568 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:51.865940 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:51.867931 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.869665 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.871427 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.872022 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.872244 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:51.882714 (+ 10470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:51.882724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:51.882747 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:51.882794 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:51.885734 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:51.886681 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:51.886691 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:51.886698 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:51.886875 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:51.887385 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:51.887388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:51.887390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:51.888020 (+ 630us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:51.888146 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:51.888784 (+ 638us) server_negotiation.cc:299] Negotiation successful
0928 14:49:51.889072 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":101,"threads_started":1}
I20260928 14:49:51.889262 31169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.863251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35543 (local address 127.13.223.65:55587)
0928 14:49:51.863485 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:51.863503 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:51.863619 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:51.866092 (+ 2473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:51.866099 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:51.866825 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:51.867681 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:51.867706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.869878 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.869882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:51.871263 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:51.871280 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.872206 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.872210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:51.872434 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:51.879759 (+ 7325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:51.879786 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:51.882212 (+ 2426us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:51.885948 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:51.885958 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:51.885980 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:51.886437 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:51.886993 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:51.886996 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:51.886998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:51.887262 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:51.888145 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:51.888151 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:51.888392 (+ 241us) client_negotiation.cc:769] Sending connection context
0928 14:49:51.888645 (+ 253us) client_negotiation.cc:240] Negotiation successful
0928 14:49:51.888996 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":265,"spinlock_wait_cycles":112896}
I20260928 14:49:51.889285 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.863344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:41647 (local address 127.13.223.65:37545)
0928 14:49:51.863702 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:51.863719 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:49:51.863836 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:49:51.866110 (+ 2274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:49:51.866113 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:49:51.866825 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:49:51.867681 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:51.867706 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.869878 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.869882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:51.871260 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:49:51.871275 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.872150 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.872155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:49:51.872500 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:51.879759 (+ 7259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:49:51.879786 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:49:51.882414 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:49:51.886255 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:51.886258 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:51.886260 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:51.886620 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:51.887084 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:49:51.887087 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:49:51.887089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:49:51.887262 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:49:51.888708 (+ 1446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:49:51.888711 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:49:51.888811 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 14:49:51.888924 (+ 113us) client_negotiation.cc:240] Negotiation successful
0928 14:49:51.889082 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":337,"spinlock_wait_cycles":18944,"thread_start_us":94,"threads_started":1}
I20260928 14:49:51.889873 31175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:51.863825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:37545 (local address 127.13.223.67:41647)
0928 14:49:51.864114 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0928 14:49:51.864136 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:49:51.864170 (+ 34us) server_negotiation.cc:407] Connection header received
0928 14:49:51.865094 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:49:51.865102 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:49:51.865568 (+ 466us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:49:51.865940 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:49:51.867931 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.869649 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.871461 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:49:51.871975 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:49:51.872176 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:49:51.882714 (+ 10538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:49:51.882724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:49:51.882747 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:49:51.882794 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:49:51.886082 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:51.886777 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:51.886782 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:51.886786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:51.886979 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:49:51.888201 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:49:51.888203 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:49:51.888205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:49:51.888591 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:49:51.888737 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 14:49:51.889086 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:49:51.889382 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
W20260928 14:49:51.892609 30786 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
W20260928 14:49:51.893008 30785 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:51.893090 30785 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9; no voters: a837633abfca4451b3e3159169b08f7b, e2558fec68df43db95e65d11d52132e4
I20260928 14:49:51.893405 31171 raft_consensus.cc:2726] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:49:52.445708 31176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:52.445211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:54183)
0928 14:49:52.445501 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:52.445576 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260928 14:49:52.643651 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:52.643038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:34711)
0928 14:49:52.643401 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:52.643534 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260928 14:49:52.863325 31178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:52.862759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:34163)
0928 14:49:52.863151 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:52.863220 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":101,"threads_started":1}
I20260928 14:49:53.447299 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:53.446791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:43609)
0928 14:49:53.447123 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:53.447190 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260928 14:49:53.645212 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:53.644697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:39729)
0928 14:49:53.645015 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:53.645087 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":92,"threads_started":1}
I20260928 14:49:53.805331 31181 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:53.805487 31181 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:53.806116 31181 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543), e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
W20260928 14:49:53.807142 30785 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
W20260928 14:49:53.807476 30786 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:53.807607 30786 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9; no voters: a837633abfca4451b3e3159169b08f7b, e2558fec68df43db95e65d11d52132e4
I20260928 14:49:53.807782 31181 raft_consensus.cc:2726] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:49:53.864913 31182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:53.864436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:44449)
0928 14:49:53.864745 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:53.864807 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":95,"threads_started":1}
I20260928 14:49:54.448756 31183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:54.448259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:43189)
0928 14:49:54.448517 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:54.448580 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260928 14:49:54.646886 31184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:54.646275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:58175)
0928 14:49:54.646699 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:54.646769 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":109,"threads_started":1}
I20260928 14:49:54.866246 31185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:54.865755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:59043)
0928 14:49:54.866064 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:54.866134 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":75,"threads_started":1}
I20260928 14:49:55.450487 31186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:55.449835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:42281)
0928 14:49:55.450312 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:55.450384 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":103,"threads_started":1}
I20260928 14:49:55.648581 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:55.647871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:56415)
0928 14:49:55.648222 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:55.648286 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":76,"threads_started":1}
I20260928 14:49:55.868124 31188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:55.867555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:43813)
0928 14:49:55.867847 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:55.867931 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260928 14:49:56.451825 31189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:56.451383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:47759)
0928 14:49:56.451663 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:56.451724 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260928 14:49:56.622308 31190 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:49:56.622458 31190 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:49:56.623029 31190 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543), e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
W20260928 14:49:56.624068 30786 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
W20260928 14:49:56.624289 30785 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
I20260928 14:49:56.624368 30785 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9; no voters: a837633abfca4451b3e3159169b08f7b, e2558fec68df43db95e65d11d52132e4
I20260928 14:49:56.624526 31190 raft_consensus.cc:2726] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:49:56.649960 31191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:56.649493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:51119)
0928 14:49:56.649786 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:56.649852 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":87,"threads_started":1}
I20260928 14:49:56.869525 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:56.869037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:42767)
0928 14:49:56.869359 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:56.869423 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":96,"threads_started":1}
I20260928 14:49:57.453244 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:57.452772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:57815)
0928 14:49:57.453059 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:57.453128 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":81,"threads_started":1}
I20260928 14:49:57.651496 31194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:57.650883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:43415)
0928 14:49:57.651274 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:57.651362 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":90,"threads_started":1}
I20260928 14:49:57.871085 31195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:57.870499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:51429)
0928 14:49:57.870892 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:57.870969 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":73,"threads_started":1}
I20260928 14:49:58.454934 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:58.454348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:48571)
0928 14:49:58.454757 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:58.454829 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":125,"threads_started":1}
I20260928 14:49:58.652885 31197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:58.652438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:57763)
0928 14:49:58.652717 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:58.652784 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":79,"threads_started":1}
I20260928 14:49:58.872705 31198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:58.872156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:34047)
0928 14:49:58.872508 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:58.872576 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
I20260928 14:49:59.456710 31199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:59.456025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:43249)
0928 14:49:59.456523 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:59.456604 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":141,"threads_started":1}
I20260928 14:49:59.654708 31200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:59.654147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:53643)
0928 14:49:59.654491 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:59.654600 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":100,"threads_started":1}
I20260928 14:49:59.874358 31201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:49:59.873782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:57585)
0928 14:49:59.874148 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:49:59.874228 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":98,"threads_started":1}
I20260928 14:50:00.458360 31202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:00.457852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:52755)
0928 14:50:00.458193 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:00.458260 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":97,"threads_started":1}
I20260928 14:50:00.656255 31203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:00.655753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:50569)
0928 14:50:00.656081 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:00.656152 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":97,"threads_started":1}
I20260928 14:50:00.687510 31204 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:00.687738 31204 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:50:00.688304 31204 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543), e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
W20260928 14:50:00.689163 30786 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
W20260928 14:50:00.689321 30785 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
I20260928 14:50:00.689400 30785 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9; no voters: a837633abfca4451b3e3159169b08f7b, e2558fec68df43db95e65d11d52132e4
I20260928 14:50:00.689617 31204 raft_consensus.cc:2726] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:00.875717 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:00.875250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:58767)
0928 14:50:00.875553 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:00.875618 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":93,"threads_started":1}
I20260928 14:50:01.460014 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:01.459454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:49887)
0928 14:50:01.459823 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:01.459895 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":128,"threads_started":1}
I20260928 14:50:01.657850 31207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:01.657311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:45261)
0928 14:50:01.657679 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:01.657750 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":87,"threads_started":1}
I20260928 14:50:01.877260 31208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:01.876798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:53209)
0928 14:50:01.877109 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:01.877163 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
I20260928 14:50:02.233233 31209 raft_consensus.cc:482] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:02.233387 31209 raft_consensus.cc:504] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a837633abfca4451b3e3159169b08f7b" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35543 } } peers { permanent_uuid: "e2558fec68df43db95e65d11d52132e4" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 41647 } } peers { permanent_uuid: "64b754c626274b11b521d4a74b28bdd9" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 44013 } }
I20260928 14:50:02.233958 31209 leader_election.cc:288] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543), e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647)
W20260928 14:50:02.234951 30785 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e2558fec68df43db95e65d11d52132e4 (127.13.223.67:41647): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
W20260928 14:50:02.235051 30786 leader_election.cc:341] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a837633abfca4451b3e3159169b08f7b (127.13.223.66:35543): Not found: Tablet not found: 65ce9ba82a6542f39e9e27696d075820
I20260928 14:50:02.235163 30786 leader_election.cc:302] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 64b754c626274b11b521d4a74b28bdd9; no voters: a837633abfca4451b3e3159169b08f7b, e2558fec68df43db95e65d11d52132e4
I20260928 14:50:02.235344 31209 raft_consensus.cc:2726] T 65ce9ba82a6542f39e9e27696d075820 P 64b754c626274b11b521d4a74b28bdd9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 14:50:02.461779 31210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:02.461256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:54817)
0928 14:50:02.461607 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:02.461679 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":90,"threads_started":1}
I20260928 14:50:02.659536 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:02.658996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:55785)
0928 14:50:02.659349 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:02.659430 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":92,"threads_started":1}
I20260928 14:50:02.878684 31212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:02.878138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:52721)
0928 14:50:02.878454 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:02.878505 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":116,"threads_started":1}
I20260928 14:50:03.463369 31213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:03.462894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:38543)
0928 14:50:03.463198 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:03.463268 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":86,"threads_started":1}
I20260928 14:50:03.661082 31214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:03.660556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:49667)
0928 14:50:03.660892 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:03.660966 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":103,"threads_started":1}
I20260928 14:50:03.880101 31215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:03.879573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:47735)
0928 14:50:03.879870 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:03.879933 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":105,"threads_started":1}
I20260928 14:50:04.465133 31216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:04.464606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:50493)
0928 14:50:04.464914 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:04.465026 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":91,"threads_started":1}
I20260928 14:50:04.662762 31217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:04.662069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:43565)
0928 14:50:04.662493 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:04.662652 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":106,"threads_started":1}
I20260928 14:50:04.881639 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:04.881114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.67:33949)
0928 14:50:04.881461 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:04.881534 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":93,"threads_started":1}
I20260928 14:50:05.467268 31219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:05.466727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.65:37569)
0928 14:50:05.467053 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:05.467165 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260928 14:50:05.664479 31220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:05.663870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44297 (local address 127.13.223.66:52025)
0928 14:50:05.664310 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:05.664375 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.126:44297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":99,"threads_started":1}
I20260928 14:50:05.860772 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30728
I20260928 14:50:05.867101 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 30900
I20260928 14:50:05.873045 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31036
2026-09-28T14:50:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59720 ms)
[----------] 30 tests from SecurityITest (265876 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:05 dist-test-slave-gkhx krb5kdc[31223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:05 dist-test-slave-gkhx krb5kdc[31223](info): set up 2 sockets
Sep 28 14:50:05 dist-test-slave-gkhx krb5kdc[31223](info): commencing operation
krb5kdc: starting...
W20260928 14:50:06.960803 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:07.107482 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42351
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:42351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:07.226737 31236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:07.227118 31236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:07.227203 31236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:07.231632 31236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:07.231747 31236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:07.231782 31236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:07.231822 31236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:07.231880 31236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:07.238396 31236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:42351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:42351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:07.239701 31236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:07.240851 31236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:07.250645 31242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.250648 31241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.250689 31244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:07.250793 31236 server_base.cc:1034] running on GCE node
I20260928 14:50:07.251773 31236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:07.252988 31236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:07.254204 31236 hybrid_clock.cc:648] HybridClock initialized: now 1790607007254157 us; error 60 us; skew 500 ppm
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:07.257578 31236 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:07.258762 31236 webserver.cc:460] Webserver started at http://127.13.223.126:44869/ using document root <none> and password file <none>
I20260928 14:50:07.259384 31236 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:07.259490 31236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:07.259739 31236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:07.262008 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "a2be26bcd8a14635a0e97fc0f01b15e3"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "9f7070e1264b6a3ebefa0c9ff42d612c"
server_key_iv: "6eed15113f5f5f49fe59beb605767df6"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.262691 31236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "a2be26bcd8a14635a0e97fc0f01b15e3"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "9f7070e1264b6a3ebefa0c9ff42d612c"
server_key_iv: "6eed15113f5f5f49fe59beb605767df6"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.267014 31236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260928 14:50:07.269793 31251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.271348 31236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:50:07.271528 31236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "a2be26bcd8a14635a0e97fc0f01b15e3"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "9f7070e1264b6a3ebefa0c9ff42d612c"
server_key_iv: "6eed15113f5f5f49fe59beb605767df6"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.271703 31236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:07.289963 31236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:07.293773 31236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:07.294045 31236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:07.302443 31236 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:42351
I20260928 14:50:07.302465 31303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:42351 every 8 connection(s)
I20260928 14:50:07.303602 31236 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:07.305426 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31236
I20260928 14:50:07.305572 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:07.305895 14205 external_mini_cluster.cc:1428] Setting key b55a5acb0c61401494d026b5de074b06
I20260928 14:50:07.307303 31304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:07.314877 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: Bootstrap starting.
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:07.318058 31304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:07.319119 31304 log.cc:824] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:07.321939 31304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: No bootstrap required, opened a new log
I20260928 14:50:07.323350 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:07.307558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46340 (local address 127.13.223.126:42351)
0928 14:50:07.308113 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:07.308132 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:07.308171 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:50:07.308858 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:07.308875 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:07.309250 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:07.309559 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:07.310701 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.311877 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.313332 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.313894 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.314041 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:07.317356 (+ 3315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:07.317364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:07.317382 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:07.317426 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:07.319730 (+ 2304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.320353 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.320360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.320366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.320502 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.320824 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.320828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.320829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.321376 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:07.321592 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:07.321886 (+ 294us) server_negotiation.cc:299] Negotiation successful
0928 14:50:07.322300 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":148,"threads_started":1}
I20260928 14:50:07.326027 31304 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } }
I20260928 14:50:07.326378 31304 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:07.326495 31304 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2be26bcd8a14635a0e97fc0f01b15e3, State: Initialized, Role: FOLLOWER
I20260928 14:50:07.326998 31304 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } }
I20260928 14:50:07.327173 31304 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:07.327268 31304 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:07.327371 31304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:07.328951 31304 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } }
I20260928 14:50:07.329331 31304 leader_election.cc:302] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2be26bcd8a14635a0e97fc0f01b15e3; no voters:
I20260928 14:50:07.329687 31304 leader_election.cc:288] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:07.329854 31309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:07.330096 31309 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [term 1 LEADER]: Becoming Leader. State: Replica: a2be26bcd8a14635a0e97fc0f01b15e3, State: Running, Role: LEADER
I20260928 14:50:07.330484 31309 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } }
I20260928 14:50:07.331099 31304 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:07.332407 31310 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } } }
I20260928 14:50:07.332544 31310 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:07.332976 31318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:07.332924 31311 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2be26bcd8a14635a0e97fc0f01b15e3. Latest consensus state: current_term: 1 leader_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2be26bcd8a14635a0e97fc0f01b15e3" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 42351 } } }
I20260928 14:50:07.334260 31311 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:07.336575 31318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:07.342945 31318 catalog_manager.cc:1347] Generated new cluster ID: d37eaa4c80cc4a71a3d8d1471a93cfd3
I20260928 14:50:07.343060 31318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:07.354379 31318 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:07.355659 31318 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:07.365778 31318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: Generated new TSK 0
I20260928 14:50:07.366798 31318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:50:07.411170 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:07.536201 31330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:07.536515 31330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:07.536562 31330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:07.541172 31330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:07.541266 31330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:07.541375 31330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:07.547032 31330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:07.548213 31330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:07.549306 31330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:07.559744 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.559738 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.559738 31338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:07.560202 31330 server_base.cc:1034] running on GCE node
I20260928 14:50:07.560712 31330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:07.561404 31330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:07.562646 31330 hybrid_clock.cc:648] HybridClock initialized: now 1790607007562614 us; error 56 us; skew 500 ppm
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:07.565732 31330 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:50:07.566768 31330 webserver.cc:460] Webserver started at http://127.13.223.65:42249/ using document root <none> and password file <none>
I20260928 14:50:07.567410 31330 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:07.567510 31330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:07.567759 31330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:07.569928 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "1882f33332ef40759dbcda71614a50aa"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "5b31ff9bf986b95f84970c1e31b61746"
server_key_iv: "460661abcf879939251b315f63865422"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.570520 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "1882f33332ef40759dbcda71614a50aa"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "5b31ff9bf986b95f84970c1e31b61746"
server_key_iv: "460661abcf879939251b315f63865422"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.574680 31330 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:07.577491 31345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.578902 31330 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:07.579070 31330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "1882f33332ef40759dbcda71614a50aa"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "5b31ff9bf986b95f84970c1e31b61746"
server_key_iv: "460661abcf879939251b315f63865422"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.579207 31330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:07.605881 31330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:07.609539 31330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:07.609777 31330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:07.610437 31330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:07.611389 31330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:07.611472 31330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.611552 31330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:07.611603 31330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.621505 31330 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:45759
I20260928 14:50:07.621533 31458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:45759 every 8 connection(s)
I20260928 14:50:07.622463 31330 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:07.629194 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31330
I20260928 14:50:07.629347 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:07.629675 14205 external_mini_cluster.cc:1428] Setting key 711bd5b1d3ac9375aebd26341b9c3d6c
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:07.639617 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:07.624552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:53171 (local address 127.13.223.126:42351)
0928 14:50:07.624744 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:07.624749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:07.626338 (+ 1589us) server_negotiation.cc:407] Connection header received
0928 14:50:07.627390 (+ 1052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:07.627394 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:07.627456 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:07.627537 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:07.628783 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.629644 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.631328 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.631649 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.631783 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:07.634957 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:07.634964 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:07.634970 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:07.635005 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:07.637018 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.637643 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.637650 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.637655 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.637761 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.638211 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.638216 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.638220 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.638485 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:07.638623 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:07.639243 (+ 620us) server_negotiation.cc:299] Negotiation successful
0928 14:50:07.639393 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:50:07.640432 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:07.625001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42351 (local address 127.13.223.65:53171)
0928 14:50:07.626051 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:07.626100 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:07.627166 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:07.627748 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:07.627755 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:07.628119 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:07.628617 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:07.628631 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.629862 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.629865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:07.631187 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:07.631200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.631872 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.631876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:07.632155 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:07.632910 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:07.632937 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:07.634770 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:07.637181 (+ 2411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:07.637187 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:07.637202 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:07.637510 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:07.637888 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:07.637891 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:07.637893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:07.638076 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:07.638667 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:07.638673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:07.638955 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 14:50:07.639138 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 14:50:07.639436 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":654,"threads_started":1}
I20260928 14:50:07.641906 31459 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42351
I20260928 14:50:07.642192 31459 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:07.642834 31459 heartbeater.cc:511] Master 127.13.223.126:42351 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:50:07.644451 31268 ts_manager.cc:194] Registered new tserver with Master: 1882f33332ef40759dbcda71614a50aa (127.13.223.65:45759)
I20260928 14:50:07.645641 31268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:53171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:50:07.664019 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:07.790827 31464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:07.791131 31464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:07.791214 31464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:07.795938 31464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:07.796039 31464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:07.796140 31464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:07.802999 31464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:07.804410 31464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:07.805681 31464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:07.816880 31470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.816830 31469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:07.816830 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:07.817414 31464 server_base.cc:1034] running on GCE node
I20260928 14:50:07.817940 31464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:07.818722 31464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:07.819921 31464 hybrid_clock.cc:648] HybridClock initialized: now 1790607007819894 us; error 39 us; skew 500 ppm
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:07.823338 31464 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:50:07.824545 31464 webserver.cc:460] Webserver started at http://127.13.223.66:36173/ using document root <none> and password file <none>
I20260928 14:50:07.825309 31464 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:07.825420 31464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:07.825714 31464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:07.828230 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "2d7020581ec04e0083af0cc81c0c0af0"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "05fc411725d1eec976676d95465ca573"
server_key_iv: "b325805a994aae8e5f5f7023409b872f"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.828922 31464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "2d7020581ec04e0083af0cc81c0c0af0"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "05fc411725d1eec976676d95465ca573"
server_key_iv: "b325805a994aae8e5f5f7023409b872f"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.833664 31464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 14:50:07.836566 31479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.838119 31464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:07.838290 31464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "2d7020581ec04e0083af0cc81c0c0af0"
format_stamp: "Formatted at 2026-09-28 14:50:07 on dist-test-slave-gkhx"
server_key: "05fc411725d1eec976676d95465ca573"
server_key_iv: "b325805a994aae8e5f5f7023409b872f"
server_key_version: "encryptionkey@0"
I20260928 14:50:07.838433 31464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:07.880100 31464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:07.884155 31464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:07.884400 31464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:07.885231 31464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:07.886202 31464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:07.886282 31464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.886374 31464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:07.886435 31464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:07.896682 31464 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:34443
I20260928 14:50:07.896705 31592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:34443 every 8 connection(s)
I20260928 14:50:07.897768 31464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:07.902480 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31464
I20260928 14:50:07.902680 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:07.903038 14205 external_mini_cluster.cc:1428] Setting key 2fd66b3d0ffbc4e35c4d47bf6c768f59
Sep 28 14:50:07 dist-test-slave-gkhx krb5kdc[31223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607007, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:50:07.915156 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:07.899916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44291 (local address 127.13.223.126:42351)
0928 14:50:07.900133 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:07.900136 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:07.901031 (+ 895us) server_negotiation.cc:407] Connection header received
0928 14:50:07.902367 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:07.902370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:07.902430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:07.902518 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:07.903956 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.904547 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.906510 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.907039 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.907177 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:07.910296 (+ 3119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:07.910301 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:07.910303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:07.910331 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:07.912100 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.912948 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.912952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.912990 (+ 38us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.913115 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:07.913606 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:07.913609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:07.913610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:07.913894 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:07.914022 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:07.914723 (+ 701us) server_negotiation.cc:299] Negotiation successful
0928 14:50:07.915023 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260928 14:50:07.916006 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:07.900290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42351 (local address 127.13.223.66:44291)
0928 14:50:07.900806 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:07.900851 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:07.902143 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:07.902738 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:07.902745 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:07.903166 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:07.903804 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:07.903821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.904693 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.904697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:07.906350 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:07.906368 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:07.907237 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:07.907241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:07.907425 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:07.908123 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:07.908182 (+ 59us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:07.910046 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:07.912263 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:07.912276 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:07.912302 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:07.912793 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:07.913240 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:07.913247 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:07.913252 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:07.913486 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:07.914050 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:07.914061 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:07.914390 (+ 329us) client_negotiation.cc:769] Sending connection context
0928 14:50:07.914672 (+ 282us) client_negotiation.cc:240] Negotiation successful
0928 14:50:07.915030 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":138,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:50:07.917416 31593 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42351
I20260928 14:50:07.917707 31593 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:07.918280 31593 heartbeater.cc:511] Master 127.13.223.126:42351 requested a full tablet report, sending...
I20260928 14:50:07.919548 31268 ts_manager.cc:194] Registered new tserver with Master: 2d7020581ec04e0083af0cc81c0c0af0 (127.13.223.66:34443)
I20260928 14:50:07.920204 31268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:44291
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:50:07.934283 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:33785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:08.057364 31598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:08.057685 31598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:08.057732 31598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:08.062129 31598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:08.062212 31598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:08.062279 31598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:08.067909 31598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:33785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:42351
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:08.069056 31598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:08.070062 31598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:08.080835 31606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.080912 31604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.081049 31603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.080883 31598 server_base.cc:1034] running on GCE node
I20260928 14:50:08.081727 31598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:08.082504 31598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:08.083734 31598 hybrid_clock.cc:648] HybridClock initialized: now 1790607008083720 us; error 30 us; skew 500 ppm
Sep 28 14:50:08 dist-test-slave-gkhx krb5kdc[31223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:08.087086 31598 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:50:08.088294 31598 webserver.cc:460] Webserver started at http://127.13.223.67:45353/ using document root <none> and password file <none>
I20260928 14:50:08.089012 31598 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:08.089134 31598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:08.089422 31598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:08.092118 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "136bf60a41094589a63a60ad652dbdb7"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "0b745e447c6425dcc4b1209e3e12514c"
server_key_iv: "52cd3a80ca243ffbf2672e2e157223d2"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.092880 31598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "136bf60a41094589a63a60ad652dbdb7"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "0b745e447c6425dcc4b1209e3e12514c"
server_key_iv: "52cd3a80ca243ffbf2672e2e157223d2"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.098067 31598 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260928 14:50:08.101325 31613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.103157 31598 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:08.103338 31598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "136bf60a41094589a63a60ad652dbdb7"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "0b745e447c6425dcc4b1209e3e12514c"
server_key_iv: "52cd3a80ca243ffbf2672e2e157223d2"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.103482 31598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:08.124054 31598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:08.127948 31598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:08.128218 31598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:08.128896 31598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:08.129848 31598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:08.129925 31598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.130000 31598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:08.130051 31598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.140540 31598 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:34973
I20260928 14:50:08.140585 31726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:34973 every 8 connection(s)
I20260928 14:50:08.141633 31598 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:08.142136 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31598
I20260928 14:50:08.142272 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:08.142650 14205 external_mini_cluster.cc:1428] Setting key 215e746e564e0ff6ee9b0ab414387b66
Sep 28 14:50:08 dist-test-slave-gkhx krb5kdc[31223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607008, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:08.158066 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.144081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:42227 (local address 127.13.223.126:42351)
0928 14:50:08.144249 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.144254 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.145231 (+ 977us) server_negotiation.cc:407] Connection header received
0928 14:50:08.146815 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.146819 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.146869 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.146946 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:08.148267 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.148821 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.150100 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.150352 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.150455 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.153524 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:08.153528 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:08.153530 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:08.153566 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:08.155194 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.155978 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.155983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.155984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.156087 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.156666 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.156670 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.156672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.156908 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:08.156999 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.157740 (+ 741us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.157888 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:50:08.159112 31729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.144481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:42351 (local address 127.13.223.67:42227)
0928 14:50:08.145046 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.145117 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.146578 (+ 1461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.147203 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.147210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.147581 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:08.148109 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.148124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.148990 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.148993 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.149941 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.149959 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.150533 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.150536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.150818 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.151578 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:08.151596 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:08.153314 (+ 1718us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:08.155385 (+ 2071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:08.155391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:08.155409 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:08.155815 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:08.156227 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:08.156231 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:08.156233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:08.156520 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:08.157030 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:08.157038 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:08.157447 (+ 409us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.157681 (+ 234us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.158077 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":117,"threads_started":1}
I20260928 14:50:08.160346 31727 heartbeater.cc:347] Connected to a master server at 127.13.223.126:42351
I20260928 14:50:08.160640 31727 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:08.161221 31727 heartbeater.cc:511] Master 127.13.223.126:42351 requested a full tablet report, sending...
I20260928 14:50:08.162328 31268 ts_manager.cc:194] Registered new tserver with Master: 136bf60a41094589a63a60ad652dbdb7 (127.13.223.67:34973)
I20260928 14:50:08.162971 31268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:42227
I20260928 14:50:08.169610 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:08.182076 31307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.171857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.13.223.126:42351)
0928 14:50:08.172112 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.172118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.172260 (+ 142us) server_negotiation.cc:407] Connection header received
0928 14:50:08.172511 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.172515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.172578 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.172690 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:08.173605 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.174238 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.175366 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.175636 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.175836 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.177576 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:08.177581 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:08.177584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:08.177627 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:08.179410 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.179968 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.179973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.179974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.180075 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:08.180425 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:08.180431 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:08.180437 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:08.181198 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:08.181378 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.181712 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.181854 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260928 14:50:08.185992 31268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:08.189296 31268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:08.205884 31738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.198962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:45759 (local address 127.0.0.1:58474)
0928 14:50:08.199975 (+ 1013us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.199987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.200086 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.200794 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.200801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.200839 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:08.201182 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.201190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202729 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.202731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.204060 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.204069 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205211 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205216 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.205355 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205410 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.205535 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.205682 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":422,"spinlock_wait_cycles":9088,"thread_start_us":107,"threads_started":1}
I20260928 14:50:08.205971 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.200203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.13.223.66:34443)
0928 14:50:08.200867 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.200886 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.200906 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:08.200969 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.200973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.201124 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.201289 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:08.201574 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202396 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.204504 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205336 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205437 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205574 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.205743 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.205840 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":138,"threads_started":1}
I20260928 14:50:08.206504 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.199124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58474 (local address 127.13.223.65:45759)
0928 14:50:08.199551 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.199558 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.200182 (+ 624us) server_negotiation.cc:407] Connection header received
0928 14:50:08.200256 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.200260 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.200455 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.200601 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:08.201326 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202541 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.204196 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205062 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205182 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205351 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.206223 (+ 872us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.206348 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":100,"threads_started":1}
I20260928 14:50:08.206737 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.199308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.13.223.67:34973)
0928 14:50:08.199835 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.199847 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.200154 (+ 307us) server_negotiation.cc:407] Connection header received
0928 14:50:08.200239 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.200244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.200455 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.200601 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:08.201337 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202542 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.204586 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205286 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205397 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205573 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.206030 (+ 457us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.206527 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":222,"threads_started":1}
I20260928 14:50:08.205920 31742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.199840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34443 (local address 127.0.0.1:41856)
0928 14:50:08.200623 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.200638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.200760 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.201275 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.201278 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.201298 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:08.201427 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.201437 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202545 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.202574 (+ 29us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.204354 (+ 1780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.204366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205447 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.205565 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205593 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.205696 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.205779 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":129,"thread_start_us":107,"threads_started":1}
I20260928 14:50:08.206363 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.198962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:34973 (local address 127.0.0.1:42566)
0928 14:50:08.199807 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.199861 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.200005 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.200794 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.200801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.200837 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:08.201182 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.201190 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.202883 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.202891 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.204426 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.204438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.205636 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.205640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.205737 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.205762 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.205834 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.205904 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"mutex_wait_us":129,"spinlock_wait_cycles":4736,"thread_start_us":96,"threads_started":1}
I20260928 14:50:08.209991 31393 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 (DEFAULT_TABLE table=test-table [id=6acf8af0b7dd41839ff05a2abda15d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:08.210132 31527 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 (DEFAULT_TABLE table=test-table [id=6acf8af0b7dd41839ff05a2abda15d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:08.210386 31661 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 (DEFAULT_TABLE table=test-table [id=6acf8af0b7dd41839ff05a2abda15d7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:08.211256 31527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8231ee8d2e498c953bb2aff0c78fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:08.211256 31393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8231ee8d2e498c953bb2aff0c78fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:08.211282 31661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8231ee8d2e498c953bb2aff0c78fc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:08.217978 31744 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: Bootstrap starting.
I20260928 14:50:08.219563 31745 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Bootstrap starting.
I20260928 14:50:08.220985 31744 tablet_bootstrap.cc:654] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:08.221834 31745 tablet_bootstrap.cc:654] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:08.221948 31744 log.cc:824] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:08.222946 31746 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Bootstrap starting.
I20260928 14:50:08.223181 31745 log.cc:824] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:08.224434 31744 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: No bootstrap required, opened a new log
I20260928 14:50:08.224674 31744 ts_tablet_manager.cc:1397] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 14:50:08.225278 31745 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: No bootstrap required, opened a new log
I20260928 14:50:08.225329 31746 tablet_bootstrap.cc:654] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:08.225464 31745 ts_tablet_manager.cc:1397] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:50:08.226673 31746 log.cc:824] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:08.228976 31746 tablet_bootstrap.cc:492] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: No bootstrap required, opened a new log
I20260928 14:50:08.229259 31746 ts_tablet_manager.cc:1397] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:50:08.229090 31744 raft_consensus.cc:348] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.229475 31744 raft_consensus.cc:374] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:08.229365 31745 raft_consensus.cc:348] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.229562 31744 raft_consensus.cc:729] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1882f33332ef40759dbcda71614a50aa, State: Initialized, Role: FOLLOWER
I20260928 14:50:08.229671 31745 raft_consensus.cc:374] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:08.229734 31745 raft_consensus.cc:729] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 136bf60a41094589a63a60ad652dbdb7, State: Initialized, Role: FOLLOWER
I20260928 14:50:08.230083 31744 consensus_queue.cc:260] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.230167 31745 consensus_queue.cc:260] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.230841 31459 heartbeater.cc:503] Master 127.13.223.126:42351 was elected leader, sending a full tablet report...
I20260928 14:50:08.231205 31744 ts_tablet_manager.cc:1428] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260928 14:50:08.231292 31745 ts_tablet_manager.cc:1428] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260928 14:50:08.231665 31727 heartbeater.cc:503] Master 127.13.223.126:42351 was elected leader, sending a full tablet report...
I20260928 14:50:08.234962 31746 raft_consensus.cc:348] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.235260 31746 raft_consensus.cc:374] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:08.235337 31746 raft_consensus.cc:729] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7020581ec04e0083af0cc81c0c0af0, State: Initialized, Role: FOLLOWER
I20260928 14:50:08.235901 31746 consensus_queue.cc:260] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.236749 31593 heartbeater.cc:503] Master 127.13.223.126:42351 was elected leader, sending a full tablet report...
I20260928 14:50:08.237104 31746 ts_tablet_manager.cc:1428] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 14:50:08.280104 31750 raft_consensus.cc:482] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:08.280308 31750 raft_consensus.cc:504] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.281415 31750 leader_election.cc:288] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 136bf60a41094589a63a60ad652dbdb7 (127.13.223.67:34973), 2d7020581ec04e0083af0cc81c0c0af0 (127.13.223.66:34443)
I20260928 14:50:08.286819 31743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.282331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:34843 (local address 127.13.223.66:34443)
0928 14:50:08.282505 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.282510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.282840 (+ 330us) server_negotiation.cc:407] Connection header received
0928 14:50:08.282950 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.282952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.282997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.283065 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:08.283343 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.283963 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.285323 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.285935 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.286064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.286117 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.286600 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.286680 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:50:08.286819 31741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.281814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:51349 (local address 127.13.223.67:34973)
0928 14:50:08.281980 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.281985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.282216 (+ 231us) server_negotiation.cc:407] Connection header received
0928 14:50:08.282385 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.282391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.282450 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.282535 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:08.282915 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.283664 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.285297 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.285935 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.286064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.286117 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.286600 (+ 483us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.286694 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 14:50:08.286919 31754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.282207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34443 (local address 127.13.223.65:34843)
0928 14:50:08.282743 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.282755 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.282835 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.283092 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.283095 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.283113 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:08.283230 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.283236 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.284074 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.284077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.285204 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.285214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.286141 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.286145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.286378 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.286466 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.286590 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.286693 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":79,"thread_start_us":104,"threads_started":1}
I20260928 14:50:08.286922 31753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.281719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:34973 (local address 127.13.223.65:51349)
0928 14:50:08.282069 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.282090 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.282258 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.282603 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.282606 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.282645 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:08.282772 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.282777 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.283789 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.283795 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.285138 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.285150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.286206 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.286210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.286418 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.286466 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.286590 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.286680 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":76,"thread_start_us":147,"threads_started":1}
I20260928 14:50:08.288049 31547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8231ee8d2e498c953bb2aff0c78fc8" candidate_uuid: "1882f33332ef40759dbcda71614a50aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7020581ec04e0083af0cc81c0c0af0" is_pre_election: true
I20260928 14:50:08.288053 31681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8231ee8d2e498c953bb2aff0c78fc8" candidate_uuid: "1882f33332ef40759dbcda71614a50aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136bf60a41094589a63a60ad652dbdb7" is_pre_election: true
I20260928 14:50:08.288378 31681 raft_consensus.cc:2454] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1882f33332ef40759dbcda71614a50aa in term 0.
I20260928 14:50:08.288379 31547 raft_consensus.cc:2454] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1882f33332ef40759dbcda71614a50aa in term 0.
I20260928 14:50:08.288910 31349 leader_election.cc:302] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 136bf60a41094589a63a60ad652dbdb7, 1882f33332ef40759dbcda71614a50aa; no voters:
I20260928 14:50:08.289234 31750 raft_consensus.cc:2781] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:08.289314 31750 raft_consensus.cc:482] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:08.289345 31750 raft_consensus.cc:3037] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:08.290436 31750 raft_consensus.cc:504] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.290865 31750 leader_election.cc:288] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [CANDIDATE]: Term 1 election: Requested vote from peers 136bf60a41094589a63a60ad652dbdb7 (127.13.223.67:34973), 2d7020581ec04e0083af0cc81c0c0af0 (127.13.223.66:34443)
I20260928 14:50:08.291493 31681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8231ee8d2e498c953bb2aff0c78fc8" candidate_uuid: "1882f33332ef40759dbcda71614a50aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "136bf60a41094589a63a60ad652dbdb7"
I20260928 14:50:08.291647 31681 raft_consensus.cc:3037] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:08.291592 31547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8231ee8d2e498c953bb2aff0c78fc8" candidate_uuid: "1882f33332ef40759dbcda71614a50aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d7020581ec04e0083af0cc81c0c0af0"
I20260928 14:50:08.291900 31547 raft_consensus.cc:3037] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:08.292975 31547 raft_consensus.cc:2454] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1882f33332ef40759dbcda71614a50aa in term 1.
I20260928 14:50:08.292979 31681 raft_consensus.cc:2454] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1882f33332ef40759dbcda71614a50aa in term 1.
I20260928 14:50:08.293377 31349 leader_election.cc:302] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1882f33332ef40759dbcda71614a50aa, 2d7020581ec04e0083af0cc81c0c0af0; no voters:
I20260928 14:50:08.293638 31750 raft_consensus.cc:2781] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:08.293920 31750 raft_consensus.cc:686] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 1 LEADER]: Becoming Leader. State: Replica: 1882f33332ef40759dbcda71614a50aa, State: Running, Role: LEADER
I20260928 14:50:08.294349 31750 consensus_queue.cc:237] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } }
I20260928 14:50:08.297868 31268 catalog_manager.cc:5495] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa reported cstate change: term changed from 0 to 1, leader changed from <none> to 1882f33332ef40759dbcda71614a50aa (127.13.223.65). New cstate: current_term: 1 leader_uuid: "1882f33332ef40759dbcda71614a50aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1882f33332ef40759dbcda71614a50aa" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 45759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:08.330916 31740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.326937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.13.223.65:45759)
0928 14:50:08.327087 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.327094 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.327112 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:08.327203 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.327206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.327264 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.327352 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:08.327590 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.328097 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.329072 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.329322 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.329424 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.329827 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:08.330308 (+ 481us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:08.330421 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.330640 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.330720 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 14:50:08.337641 31547 raft_consensus.cc:1261] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 1 FOLLOWER]: Refusing update from remote peer 1882f33332ef40759dbcda71614a50aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:08.337694 31681 raft_consensus.cc:1261] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 1 FOLLOWER]: Refusing update from remote peer 1882f33332ef40759dbcda71614a50aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:08.338457 31750 consensus_queue.cc:1035] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d7020581ec04e0083af0cc81c0c0af0" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:08.338712 31755 consensus_queue.cc:1035] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "136bf60a41094589a63a60ad652dbdb7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 34973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:08.346452 31758 mvcc.cc:205] Tried to move back new op lower bound from 7334326306142228480 to 7334326305976844288. Current Snapshot: MvccSnapshot[applied={T|T < 7334326306142228480}]
I20260928 14:50:08.347612 31760 mvcc.cc:205] Tried to move back new op lower bound from 7334326306142228480 to 7334326305976844288. Current Snapshot: MvccSnapshot[applied={T|T < 7334326306142228480}]
I20260928 14:50:08.356817 31267 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:08.357105 31267 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46348:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:08.360087 31267 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2be26bcd8a14635a0e97fc0f01b15e3: Sending DeleteTablet for 3 replicas of tablet 5c8231ee8d2e498c953bb2aff0c78fc8
I20260928 14:50:08.361240 31661 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:42566
I20260928 14:50:08.361326 31393 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:58474
I20260928 14:50:08.361356 31527 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8231ee8d2e498c953bb2aff0c78fc8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:08 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:41856
I20260928 14:50:08.361845 31768 tablet_replica.cc:330] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa: stopping tablet replica
I20260928 14:50:08.362193 31768 raft_consensus.cc:2229] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:08.362314 31770 tablet_replica.cc:330] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: stopping tablet replica
I20260928 14:50:08.362833 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31330
I20260928 14:50:08.362701 31768 raft_consensus.cc:2258] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 1882f33332ef40759dbcda71614a50aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:08.365315 31770 raft_consensus.cc:2229] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:08.365741 31770 raft_consensus.cc:2258] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:08.366289 31769 tablet_replica.cc:330] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: stopping tablet replica
I20260928 14:50:08.366771 31769 raft_consensus.cc:2229] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:08.367144 31769 raft_consensus.cc:2258] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:08.367712 31770 ts_tablet_manager.cc:1905] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:08.369024 31769 ts_tablet_manager.cc:1905] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 2d7020581ec04e0083af0cc81c0c0af0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:08.369979 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31464
W20260928 14:50:08.369959 31253 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:45759 (error 108)
I20260928 14:50:08.372262 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.371874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:45759 (local address 127.0.0.1:58492)
0928 14:50:08.372057 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.372145 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:45759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260928 14:50:08.372427 31770 ts_tablet_manager.cc:1918] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:08.372558 31770 log.cc:1196] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/5c8231ee8d2e498c953bb2aff0c78fc8
W20260928 14:50:08.372619 31253 catalog_manager.cc:4584] TS 1882f33332ef40759dbcda71614a50aa (127.13.223.65:45759): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5c8231ee8d2e498c953bb2aff0c78fc8: Network error: Client connection negotiation failed: client connection to 127.13.223.65:45759: connect: Connection refused (error 111)
I20260928 14:50:08.373034 31770 ts_tablet_manager.cc:1939] T 5c8231ee8d2e498c953bb2aff0c78fc8 P 136bf60a41094589a63a60ad652dbdb7: Deleting consensus metadata
I20260928 14:50:08.374459 31255 catalog_manager.cc:4841] TS 136bf60a41094589a63a60ad652dbdb7 (127.13.223.67:34973): tablet 5c8231ee8d2e498c953bb2aff0c78fc8 (table test-table [id=6acf8af0b7dd41839ff05a2abda15d7e]) successfully deleted
I20260928 14:50:08.377099 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31598
I20260928 14:50:08.378969 31739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.378685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34443 (local address 127.0.0.1:41870)
0928 14:50:08.378818 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.378883 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:34443: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 14:50:08.383206 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31236
2026-09-28T14:50:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2510 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T14:50:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:08Z Disabled control of system clock
I20260928 14:50:08.417572 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40727
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:08.533918 31775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:08.534211 31775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:08.534248 31775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:08.538472 31775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:08.538570 31775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:08.538587 31775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:08.538605 31775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:08.538621 31775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:08.543793 31775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:40727
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:08.544981 31775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:08.546393 31775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:08.555766 31783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.555766 31781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.555861 31775 server_base.cc:1034] running on GCE node
W20260928 14:50:08.555794 31780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.556833 31775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:08.558146 31775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:08.559355 31775 hybrid_clock.cc:648] HybridClock initialized: now 1790607008559335 us; error 37 us; skew 500 ppm
I20260928 14:50:08.562404 31775 webserver.cc:460] Webserver started at http://127.13.223.126:41855/ using document root <none> and password file <none>
I20260928 14:50:08.563169 31775 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:08.563249 31775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:08.563443 31775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:08.566001 31775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "da78fd1f29d843789d40a358ba195bd9"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "d7e33157e3839235d2388ba1d8789935"
server_key_iv: "e993023dbc62bcf6722b78f5ca22c15d"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.566780 31775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "da78fd1f29d843789d40a358ba195bd9"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "d7e33157e3839235d2388ba1d8789935"
server_key_iv: "e993023dbc62bcf6722b78f5ca22c15d"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.571223 31775 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:08.574256 31789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.575831 31775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:08.575990 31775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "da78fd1f29d843789d40a358ba195bd9"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "d7e33157e3839235d2388ba1d8789935"
server_key_iv: "e993023dbc62bcf6722b78f5ca22c15d"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.576184 31775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:08.595656 31775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:08.596493 31775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:08.596728 31775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:08.604947 31775 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:40727
I20260928 14:50:08.604974 31841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:40727 every 8 connection(s)
I20260928 14:50:08.606199 31775 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:08.610136 31842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:08.614984 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31775
I20260928 14:50:08.615167 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:08.615597 14205 external_mini_cluster.cc:1428] Setting key fdc91b7dc9a9b81ff812a18bf252b31f
I20260928 14:50:08.617830 31842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: Bootstrap starting.
I20260928 14:50:08.620709 31842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:08.621618 31842 log.cc:824] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:08.623997 31842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: No bootstrap required, opened a new log
I20260928 14:50:08.627146 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.617383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.13.223.126:40727)
0928 14:50:08.618242 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.618270 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.618331 (+ 61us) server_negotiation.cc:407] Connection header received
0928 14:50:08.619190 (+ 859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.619212 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.619698 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.620120 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:08.620633 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.621905 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.623015 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.623567 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.623772 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.624771 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:08.624777 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:08.624793 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:08.625260 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:08.625327 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:08.625350 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:08.625486 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.625660 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.625996 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":177,"threads_started":1}
I20260928 14:50:08.628010 31842 raft_consensus.cc:348] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } }
I20260928 14:50:08.628352 31842 raft_consensus.cc:374] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:08.628449 31842 raft_consensus.cc:729] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da78fd1f29d843789d40a358ba195bd9, State: Initialized, Role: FOLLOWER
I20260928 14:50:08.629041 31842 consensus_queue.cc:260] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } }
I20260928 14:50:08.629202 31842 raft_consensus.cc:388] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:08.629303 31842 raft_consensus.cc:482] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:08.629459 31842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:08.630999 31842 raft_consensus.cc:504] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } }
I20260928 14:50:08.631438 31842 leader_election.cc:302] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da78fd1f29d843789d40a358ba195bd9; no voters:
I20260928 14:50:08.631839 31842 leader_election.cc:288] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:08.632118 31847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:08.632442 31847 raft_consensus.cc:686] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [term 1 LEADER]: Becoming Leader. State: Replica: da78fd1f29d843789d40a358ba195bd9, State: Running, Role: LEADER
I20260928 14:50:08.632781 31847 consensus_queue.cc:237] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } }
I20260928 14:50:08.633323 31842 sys_catalog.cc:564] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:08.634760 31849 sys_catalog.cc:455] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da78fd1f29d843789d40a358ba195bd9. Latest consensus state: current_term: 1 leader_uuid: "da78fd1f29d843789d40a358ba195bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } } }
I20260928 14:50:08.634912 31849 sys_catalog.cc:458] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:08.635334 31848 sys_catalog.cc:455] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da78fd1f29d843789d40a358ba195bd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da78fd1f29d843789d40a358ba195bd9" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 40727 } } }
I20260928 14:50:08.635373 31856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:08.635462 31848 sys_catalog.cc:458] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:08.639266 31856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:08.646438 31856 catalog_manager.cc:1347] Generated new cluster ID: ccaa38e4f54640398657c3f5c3cfa460
I20260928 14:50:08.646589 31856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:08.664271 31856 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:08.665781 31856 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:08.683326 31856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: Generated new TSK 0
I20260928 14:50:08.684185 31856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:08.695005 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--builtin_ntp_servers=127.13.223.84:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:08.828063 31866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:08.828415 31866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:08.828500 31866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:08.832834 31866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:08.832945 31866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:08.833050 31866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:08.838698 31866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:08.839926 31866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:08.841425 31866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:08.852356 31871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.852358 31874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:08.852407 31872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:08.852686 31866 server_base.cc:1034] running on GCE node
I20260928 14:50:08.853181 31866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:08.853984 31866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:08.855261 31866 hybrid_clock.cc:648] HybridClock initialized: now 1790607008855225 us; error 47 us; skew 500 ppm
I20260928 14:50:08.857816 31866 webserver.cc:460] Webserver started at http://127.13.223.65:34809/ using document root <none> and password file <none>
I20260928 14:50:08.858668 31866 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:08.858788 31866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:08.859066 31866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:08.861516 31866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "218947a925e349079a05a08cc47ae176"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "6eec12a3cafe50950d82ec6ae5a90ef7"
server_key_iv: "4a0aa003ce7139591036edd8548bdef9"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.862208 31866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "218947a925e349079a05a08cc47ae176"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "6eec12a3cafe50950d82ec6ae5a90ef7"
server_key_iv: "4a0aa003ce7139591036edd8548bdef9"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.868013 31866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 14:50:08.871217 31880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.872675 31866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:50:08.872855 31866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "218947a925e349079a05a08cc47ae176"
format_stamp: "Formatted at 2026-09-28 14:50:08 on dist-test-slave-gkhx"
server_key: "6eec12a3cafe50950d82ec6ae5a90ef7"
server_key_iv: "4a0aa003ce7139591036edd8548bdef9"
server_key_version: "encryptionkey@0"
I20260928 14:50:08.873093 31866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:08.896283 31866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:08.897127 31866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:08.897354 31866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:08.898012 31866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:08.899187 31866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:08.899266 31866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.899350 31866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:08.899403 31866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:08.909330 31866 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:39889
I20260928 14:50:08.909353 31993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:39889 every 8 connection(s)
I20260928 14:50:08.910357 31866 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:08.914340 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31866
I20260928 14:50:08.914458 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:08.914776 14205 external_mini_cluster.cc:1428] Setting key 44c63889e0d47abf27a8c640cf8324dd
I20260928 14:50:08.917201 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--builtin_ntp_servers=127.13.223.84:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:08.922878 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.912603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:39515 (local address 127.13.223.126:40727)
0928 14:50:08.912760 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:08.912764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:08.914094 (+ 1330us) server_negotiation.cc:407] Connection header received
0928 14:50:08.915910 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:08.915914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:08.915959 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:08.916062 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:08.917484 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.918071 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.920158 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.920418 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.920534 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.922168 (+ 1634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:08.922172 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:08.922174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:08.922188 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:08.922211 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:08.922219 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:08.922283 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:08.922681 (+ 398us) server_negotiation.cc:299] Negotiation successful
0928 14:50:08.922750 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:50:08.924295 31996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:08.913057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40727 (local address 127.13.223.65:39515)
0928 14:50:08.913849 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:08.913899 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:08.915694 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:08.916376 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:08.916388 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:08.916519 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:08.917194 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.917212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.918239 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.918246 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.919966 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:08.919986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:08.920644 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:08.920652 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:08.920951 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:08.921759 (+ 808us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:08.921764 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:08.921837 (+ 73us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:08.921954 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:08.922323 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:08.922337 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:08.922348 (+ 11us) client_negotiation.cc:769] Sending connection context
0928 14:50:08.922532 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 14:50:08.922979 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":144,"threads_started":1}
I20260928 14:50:08.925967 31994 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40727
I20260928 14:50:08.926324 31994 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:08.927191 31994 heartbeater.cc:511] Master 127.13.223.126:40727 requested a full tablet report, sending...
I20260928 14:50:08.929144 31806 ts_manager.cc:194] Registered new tserver with Master: 218947a925e349079a05a08cc47ae176 (127.13.223.65:39889)
I20260928 14:50:08.930428 31806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.65:39515
W20260928 14:50:09.041744 31997 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:09.042155 31997 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:09.042239 31997 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:09.046834 31997 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:09.046948 31997 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:09.047053 31997 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:09.052295 31997 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31997
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:09.053541 31997 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:09.055132 31997 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:09.065811 32002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.065929 31997 server_base.cc:1034] running on GCE node
W20260928 14:50:09.065802 32005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.065804 32003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.066606 31997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:09.067317 31997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:09.068501 31997 hybrid_clock.cc:648] HybridClock initialized: now 1790607009068454 us; error 58 us; skew 500 ppm
I20260928 14:50:09.070745 31997 webserver.cc:460] Webserver started at http://127.13.223.66:36655/ using document root <none> and password file <none>
I20260928 14:50:09.071372 31997 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:09.071440 31997 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:09.071691 31997 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:09.074044 31997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "a47bd8425cf4447b825521669499b85e"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "35298b36e17e9279f9866dbf8f9ed1d0"
server_key_iv: "c7b49367289585f2086d282cb619fc18"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.074743 31997 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "a47bd8425cf4447b825521669499b85e"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "35298b36e17e9279f9866dbf8f9ed1d0"
server_key_iv: "c7b49367289585f2086d282cb619fc18"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.078967 31997 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:09.081928 32011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.083740 31997 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:09.083906 31997 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "a47bd8425cf4447b825521669499b85e"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "35298b36e17e9279f9866dbf8f9ed1d0"
server_key_iv: "c7b49367289585f2086d282cb619fc18"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.084019 31997 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:09.111662 31997 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:09.112409 31997 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:09.112591 31997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:09.113184 31997 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:09.114176 31997 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:09.114226 31997 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.114269 31997 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:09.114283 31997 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.124712 31997 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:34803
I20260928 14:50:09.124747 32124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:34803 every 8 connection(s)
I20260928 14:50:09.125712 31997 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:09.135818 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 31997
I20260928 14:50:09.135991 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:09.136366 14205 external_mini_cluster.cc:1428] Setting key 1f03a11ccb54b853d3ac4795a5b4fbfa
I20260928 14:50:09.136834 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.127825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:60495 (local address 127.13.223.126:40727)
0928 14:50:09.127970 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.127974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.129142 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 14:50:09.130834 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.130839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.130907 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.131027 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.132103 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.132722 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.134266 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.134519 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.134683 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.136221 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.136226 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.136230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:09.136250 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.136279 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:09.136291 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.136378 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.136585 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.136670 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:50:09.137945 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.128193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40727 (local address 127.13.223.66:60495)
0928 14:50:09.128955 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.129009 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.130529 (+ 1520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.131255 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.131263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.131363 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:09.131910 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.131924 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.132909 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.132915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.134119 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.134132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.134759 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.134763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.134989 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.135836 (+ 847us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:09.135838 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:09.135887 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:09.135992 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:09.136385 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:09.136394 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:09.136399 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.136549 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.136874 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":122,"threads_started":1}
I20260928 14:50:09.138504 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--builtin_ntp_servers=127.13.223.84:41869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:09.139742 32125 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40727
I20260928 14:50:09.140057 32125 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:09.140652 32125 heartbeater.cc:511] Master 127.13.223.126:40727 requested a full tablet report, sending...
I20260928 14:50:09.141911 31806 ts_manager.cc:194] Registered new tserver with Master: a47bd8425cf4447b825521669499b85e (127.13.223.66:34803)
I20260928 14:50:09.142896 31806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.66:60495
W20260928 14:50:09.260905 32128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:09.261243 32128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:09.261353 32128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:09.266180 32128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:09.266252 32128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:09.266314 32128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:09.271579 32128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:41869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:40727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:09.272707 32128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:09.274119 32128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:09.284925 32134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.284929 32133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:09.284929 32136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:09.285434 32128 server_base.cc:1034] running on GCE node
I20260928 14:50:09.285898 32128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:09.286710 32128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:09.287905 32128 hybrid_clock.cc:648] HybridClock initialized: now 1790607009287880 us; error 42 us; skew 500 ppm
I20260928 14:50:09.290149 32128 webserver.cc:460] Webserver started at http://127.13.223.67:34313/ using document root <none> and password file <none>
I20260928 14:50:09.290869 32128 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:09.290967 32128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:09.291220 32128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:09.293418 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "315d61f8fbc1410a8dd3587513c932ff"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "51c78037a4782e8b1bc0fa5329aaa087"
server_key_iv: "967c477658ed0aac82be36b95b85c90a"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.294059 32128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "315d61f8fbc1410a8dd3587513c932ff"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "51c78037a4782e8b1bc0fa5329aaa087"
server_key_iv: "967c477658ed0aac82be36b95b85c90a"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.298363 32128 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 14:50:09.301630 32142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.303375 32128 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:09.303584 32128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "315d61f8fbc1410a8dd3587513c932ff"
format_stamp: "Formatted at 2026-09-28 14:50:09 on dist-test-slave-gkhx"
server_key: "51c78037a4782e8b1bc0fa5329aaa087"
server_key_iv: "967c477658ed0aac82be36b95b85c90a"
server_key_version: "encryptionkey@0"
I20260928 14:50:09.303748 32128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:09.319497 32128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:09.320349 32128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:09.320600 32128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:09.321331 32128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:09.322474 32128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:09.322564 32128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.322701 32128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:09.322758 32128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:09.333571 32128 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:46069
I20260928 14:50:09.333596 32255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:46069 every 8 connection(s)
I20260928 14:50:09.334782 32128 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:09.336460 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 32128
I20260928 14:50:09.336570 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:09.336935 14205 external_mini_cluster.cc:1428] Setting key 7bedaa1d8e5204a131ead07903808aad
I20260928 14:50:09.345948 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.336918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:41403 (local address 127.13.223.126:40727)
0928 14:50:09.337087 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.337093 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.338691 (+ 1598us) server_negotiation.cc:407] Connection header received
0928 14:50:09.340307 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.340311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.340372 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.340460 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.341517 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.342130 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.343626 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.343905 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.344042 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.345355 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.345359 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.345362 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:09.345378 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.345409 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:09.345418 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.345510 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.345695 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.345781 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:50:09.347030 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.337522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:40727 (local address 127.13.223.67:41403)
0928 14:50:09.338493 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.338597 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.340066 (+ 1469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.340689 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.340697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.340798 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:09.341357 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.341371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.342294 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.342298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.343465 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.343477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.344089 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.344094 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.344299 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.345042 (+ 743us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:09.345044 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:09.345093 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:09.345171 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:09.345511 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:09.345524 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:09.345543 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.345682 (+ 139us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.346031 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":163,"threads_started":1}
I20260928 14:50:09.348542 32256 heartbeater.cc:347] Connected to a master server at 127.13.223.126:40727
I20260928 14:50:09.348831 32256 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:09.349527 32256 heartbeater.cc:511] Master 127.13.223.126:40727 requested a full tablet report, sending...
I20260928 14:50:09.350898 31806 ts_manager.cc:194] Registered new tserver with Master: 315d61f8fbc1410a8dd3587513c932ff (127.13.223.67:46069)
I20260928 14:50:09.351503 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:09.351624 31806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.67:41403
I20260928 14:50:09.358359 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.353931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40694 (local address 127.13.223.126:40727)
0928 14:50:09.354083 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.354089 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.354284 (+ 195us) server_negotiation.cc:407] Connection header received
0928 14:50:09.354395 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.354397 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.354439 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.354513 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:09.354910 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.355451 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.356519 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.356762 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.356852 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.357808 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:09.357812 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:09.357814 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:09.357828 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:09.357848 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:09.357856 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:09.357919 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.358090 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.358149 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:50:09.362274 31806 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:09.365813 31806 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:09.383252 32270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.376434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:39889 (local address 127.0.0.1:57646)
0928 14:50:09.377543 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.377603 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.377752 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.378330 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.378335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.378357 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:09.378538 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.378565 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.380199 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.380202 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.381391 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.381401 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.382603 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.382607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.382742 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.382795 (+ 53us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.382919 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.383073 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":20,"thread_start_us":145,"threads_started":1}
I20260928 14:50:09.383872 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.376710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.13.223.65:39889)
0928 14:50:09.377325 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.377334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.377693 (+ 359us) server_negotiation.cc:407] Connection header received
0928 14:50:09.377898 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.377906 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.378140 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.378305 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:09.378706 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.379985 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.381541 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.382422 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.382568 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.382810 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.383341 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.383468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":91,"threads_started":1}
I20260928 14:50:09.383912 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.376434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:46069 (local address 127.0.0.1:58640)
0928 14:50:09.377825 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.377843 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.377934 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.378567 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.378571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.378589 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:09.378693 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.378698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.380116 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.380119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.382043 (+ 1924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.382052 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.383298 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.383304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.383536 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.383579 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.383678 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.383766 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1313,"mutex_wait_us":668,"thread_start_us":106,"threads_started":1}
I20260928 14:50:09.384163 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.376687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34803 (local address 127.0.0.1:52850)
0928 14:50:09.377543 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.377603 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.377752 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.379197 (+ 1445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.379200 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.379219 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:09.379304 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.379311 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.380689 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.380692 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.381695 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.381705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.382890 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.382893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.382985 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.383020 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.383089 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.383168 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"mutex_wait_us":23,"spinlock_wait_cycles":9984,"thread_start_us":124,"threads_started":1}
I20260928 14:50:09.384658 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.377270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.13.223.67:46069)
0928 14:50:09.378080 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.378091 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.378116 (+ 25us) server_negotiation.cc:407] Connection header received
0928 14:50:09.378189 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.378199 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.378380 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.378598 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:09.378787 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.379967 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.382208 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.382942 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.383036 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.383231 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.384288 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.384450 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":164,"threads_started":1}
I20260928 14:50:09.385141 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.377687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52850 (local address 127.13.223.66:34803)
0928 14:50:09.378720 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.378734 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.378760 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:50:09.378835 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.378840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.379044 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.379197 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:09.379434 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.380569 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.381834 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.382648 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.382747 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.384768 (+ 2021us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.384836 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.384972 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":921,"thread_start_us":124,"threads_started":1}
I20260928 14:50:09.387420 31928 tablet_service.cc:1467] Processing CreateTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e (DEFAULT_TABLE table=test-table [id=b6a0b7bea6754e57a7b7cbd72f9afe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:09.387818 32190 tablet_service.cc:1467] Processing CreateTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e (DEFAULT_TABLE table=test-table [id=b6a0b7bea6754e57a7b7cbd72f9afe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:09.388149 32059 tablet_service.cc:1467] Processing CreateTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e (DEFAULT_TABLE table=test-table [id=b6a0b7bea6754e57a7b7cbd72f9afe56]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:09.388787 31928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a14dfc7cc4de1b672753d3d023d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:09.389014 32190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a14dfc7cc4de1b672753d3d023d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:09.389181 32059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996a14dfc7cc4de1b672753d3d023d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:09.396263 32273 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Bootstrap starting.
I20260928 14:50:09.398353 32274 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Bootstrap starting.
I20260928 14:50:09.399279 32273 tablet_bootstrap.cc:654] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:09.400085 32273 log.cc:824] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:09.400641 32274 tablet_bootstrap.cc:654] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:09.401469 32274 log.cc:824] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:09.402333 32273 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: No bootstrap required, opened a new log
I20260928 14:50:09.402284 32275 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Bootstrap starting.
I20260928 14:50:09.402606 32273 ts_tablet_manager.cc:1397] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:50:09.403924 32274 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: No bootstrap required, opened a new log
I20260928 14:50:09.404143 32274 ts_tablet_manager.cc:1397] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:50:09.405063 32275 tablet_bootstrap.cc:654] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:09.406036 32275 log.cc:824] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:09.406440 32273 raft_consensus.cc:348] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.406793 32273 raft_consensus.cc:374] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:09.406854 32273 raft_consensus.cc:729] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 218947a925e349079a05a08cc47ae176, State: Initialized, Role: FOLLOWER
I20260928 14:50:09.407334 32273 consensus_queue.cc:260] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.408110 31994 heartbeater.cc:503] Master 127.13.223.126:40727 was elected leader, sending a full tablet report...
I20260928 14:50:09.408456 32273 ts_tablet_manager.cc:1428] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:50:09.409080 32274 raft_consensus.cc:348] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.409407 32274 raft_consensus.cc:374] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:09.409502 32274 raft_consensus.cc:729] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a47bd8425cf4447b825521669499b85e, State: Initialized, Role: FOLLOWER
I20260928 14:50:09.410076 32274 consensus_queue.cc:260] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.411072 32275 tablet_bootstrap.cc:492] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: No bootstrap required, opened a new log
I20260928 14:50:09.411319 32275 ts_tablet_manager.cc:1397] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 14:50:09.411360 32274 ts_tablet_manager.cc:1428] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 14:50:09.411680 32125 heartbeater.cc:503] Master 127.13.223.126:40727 was elected leader, sending a full tablet report...
W20260928 14:50:09.411680 31995 tablet.cc:2369] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:09.416623 32275 raft_consensus.cc:348] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.416909 32275 raft_consensus.cc:374] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:09.417074 32275 raft_consensus.cc:729] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315d61f8fbc1410a8dd3587513c932ff, State: Initialized, Role: FOLLOWER
I20260928 14:50:09.417526 32275 consensus_queue.cc:260] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.418314 32256 heartbeater.cc:503] Master 127.13.223.126:40727 was elected leader, sending a full tablet report...
I20260928 14:50:09.418627 32275 ts_tablet_manager.cc:1428] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 14:50:09.459774 32279 raft_consensus.cc:482] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:09.460024 32279 raft_consensus.cc:504] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.461331 32279 leader_election.cc:288] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315d61f8fbc1410a8dd3587513c932ff (127.13.223.67:46069), a47bd8425cf4447b825521669499b85e (127.13.223.66:34803)
I20260928 14:50:09.468068 32272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.461936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:49479 (local address 127.13.223.66:34803)
0928 14:50:09.462138 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.462143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.462282 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:50:09.462652 (+ 370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.462655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.462717 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.462825 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:09.463334 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.464243 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.466377 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.467124 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.467311 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.467373 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.467806 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.467910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 14:50:09.468068 32271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.461843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:54611 (local address 127.13.223.67:46069)
0928 14:50:09.462022 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.462028 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.462284 (+ 256us) server_negotiation.cc:407] Connection header received
0928 14:50:09.462652 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.462655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.462715 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.462826 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:09.463334 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.464257 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.466377 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.467200 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.467321 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.467373 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.467806 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.467910 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:50:09.468135 32283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.461705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:34803 (local address 127.13.223.65:49479)
0928 14:50:09.462204 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.462220 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.462334 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.462902 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.462906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.462939 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:09.463174 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.463184 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.464435 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.464441 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.466182 (+ 1741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.466199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.467361 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.467366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.467560 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.467655 (+ 95us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.467751 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.467896 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":352,"spinlock_wait_cycles":13440,"thread_start_us":105,"threads_started":1}
I20260928 14:50:09.468140 32282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.461692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:46069 (local address 127.13.223.65:54611)
0928 14:50:09.462087 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.462105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:09.462334 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:09.462902 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:09.462906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:09.462940 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:09.463137 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.463149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.464487 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.464490 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.466182 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:09.466199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.467361 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.467366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:09.467565 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.467645 (+ 80us) client_negotiation.cc:769] Sending connection context
0928 14:50:09.467751 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 14:50:09.467897 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":186,"spinlock_wait_cycles":28544,"thread_start_us":77,"threads_started":1}
I20260928 14:50:09.469174 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a14dfc7cc4de1b672753d3d023d3e" candidate_uuid: "218947a925e349079a05a08cc47ae176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d61f8fbc1410a8dd3587513c932ff" is_pre_election: true
I20260928 14:50:09.469308 32079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a14dfc7cc4de1b672753d3d023d3e" candidate_uuid: "218947a925e349079a05a08cc47ae176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47bd8425cf4447b825521669499b85e" is_pre_election: true
I20260928 14:50:09.469585 32210 raft_consensus.cc:2454] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218947a925e349079a05a08cc47ae176 in term 0.
I20260928 14:50:09.469591 32079 raft_consensus.cc:2454] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 218947a925e349079a05a08cc47ae176 in term 0.
I20260928 14:50:09.470402 31882 leader_election.cc:302] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 218947a925e349079a05a08cc47ae176, 315d61f8fbc1410a8dd3587513c932ff, a47bd8425cf4447b825521669499b85e; no voters:
I20260928 14:50:09.470880 32279 raft_consensus.cc:2781] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:09.471033 32279 raft_consensus.cc:482] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:09.471107 32279 raft_consensus.cc:3037] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:09.472366 32279 raft_consensus.cc:504] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.472810 32279 leader_election.cc:288] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [CANDIDATE]: Term 1 election: Requested vote from peers 315d61f8fbc1410a8dd3587513c932ff (127.13.223.67:46069), a47bd8425cf4447b825521669499b85e (127.13.223.66:34803)
I20260928 14:50:09.473603 32210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a14dfc7cc4de1b672753d3d023d3e" candidate_uuid: "218947a925e349079a05a08cc47ae176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315d61f8fbc1410a8dd3587513c932ff"
I20260928 14:50:09.473675 32079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "996a14dfc7cc4de1b672753d3d023d3e" candidate_uuid: "218947a925e349079a05a08cc47ae176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a47bd8425cf4447b825521669499b85e"
I20260928 14:50:09.473801 32079 raft_consensus.cc:3037] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:09.473801 32210 raft_consensus.cc:3037] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:09.475349 32079 raft_consensus.cc:2454] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218947a925e349079a05a08cc47ae176 in term 1.
I20260928 14:50:09.475349 32210 raft_consensus.cc:2454] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 218947a925e349079a05a08cc47ae176 in term 1.
I20260928 14:50:09.475860 31882 leader_election.cc:302] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 218947a925e349079a05a08cc47ae176, 315d61f8fbc1410a8dd3587513c932ff, a47bd8425cf4447b825521669499b85e; no voters:
I20260928 14:50:09.476208 32279 raft_consensus.cc:2781] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:09.476547 32279 raft_consensus.cc:686] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 1 LEADER]: Becoming Leader. State: Replica: 218947a925e349079a05a08cc47ae176, State: Running, Role: LEADER
I20260928 14:50:09.476976 32279 consensus_queue.cc:237] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } }
I20260928 14:50:09.480842 31806 catalog_manager.cc:5495] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 reported cstate change: term changed from 0 to 1, leader changed from <none> to 218947a925e349079a05a08cc47ae176 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "218947a925e349079a05a08cc47ae176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "218947a925e349079a05a08cc47ae176" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 39889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:09.509317 32268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.504867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57662 (local address 127.13.223.65:39889)
0928 14:50:09.505096 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:09.505101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:09.505117 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:50:09.505202 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:09.505204 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:09.505281 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:09.505386 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:09.505650 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.506277 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.507440 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:09.507702 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:09.507861 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:09.508247 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:09.508744 (+ 497us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:09.508843 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:09.509032 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:50:09.509093 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260928 14:50:09.516360 32079 raft_consensus.cc:1261] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 1 FOLLOWER]: Refusing update from remote peer 218947a925e349079a05a08cc47ae176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:09.516360 32210 raft_consensus.cc:1261] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 1 FOLLOWER]: Refusing update from remote peer 218947a925e349079a05a08cc47ae176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:09.517203 32279 consensus_queue.cc:1035] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315d61f8fbc1410a8dd3587513c932ff" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:09.517418 32284 consensus_queue.cc:1035] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a47bd8425cf4447b825521669499b85e" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 34803 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:09.528734 32289 mvcc.cc:205] Tried to move back new op lower bound from 7334326310969933824 to 7334326310821195776. Current Snapshot: MvccSnapshot[applied={T|T < 7334326310969933824}]
I20260928 14:50:09.536453 31806 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:09.536686 31806 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:09.539656 31806 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da78fd1f29d843789d40a358ba195bd9: Sending DeleteTablet for 3 replicas of tablet 996a14dfc7cc4de1b672753d3d023d3e
I20260928 14:50:09.540700 32190 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:09 UTC) from {username='slave'} at 127.0.0.1:58640
I20260928 14:50:09.540721 32059 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:09 UTC) from {username='slave'} at 127.0.0.1:52850
I20260928 14:50:09.541308 32298 tablet_replica.cc:330] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: stopping tablet replica
I20260928 14:50:09.541337 31928 tablet_service.cc:1514] Processing DeleteTablet for tablet 996a14dfc7cc4de1b672753d3d023d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:09 UTC) from {username='slave'} at 127.0.0.1:57646
I20260928 14:50:09.541750 32298 raft_consensus.cc:2229] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:09.541937 32299 tablet_replica.cc:330] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176: stopping tablet replica
I20260928 14:50:09.542165 32298 raft_consensus.cc:2258] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:09.542382 32299 raft_consensus.cc:2229] T 996a14dfc7cc4de1b672753d3d023d3e P 218947a925e349079a05a08cc47ae176 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:09.542718 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31866
I20260928 14:50:09.544245 32298 ts_tablet_manager.cc:1905] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:09.544574 32297 tablet_replica.cc:330] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: stopping tablet replica
I20260928 14:50:09.544956 32297 raft_consensus.cc:2229] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:09.545305 32297 raft_consensus.cc:2258] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:09.546876 32297 ts_tablet_manager.cc:1905] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:09.548465 32298 ts_tablet_manager.cc:1918] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:09.548604 32298 log.cc:1196] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/996a14dfc7cc4de1b672753d3d023d3e
I20260928 14:50:09.548974 32298 ts_tablet_manager.cc:1939] T 996a14dfc7cc4de1b672753d3d023d3e P a47bd8425cf4447b825521669499b85e: Deleting consensus metadata
I20260928 14:50:09.549295 32297 ts_tablet_manager.cc:1918] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:09.549415 32297 log.cc:1196] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/996a14dfc7cc4de1b672753d3d023d3e
I20260928 14:50:09.549685 32297 ts_tablet_manager.cc:1939] T 996a14dfc7cc4de1b672753d3d023d3e P 315d61f8fbc1410a8dd3587513c932ff: Deleting consensus metadata
I20260928 14:50:09.550724 31793 catalog_manager.cc:4841] TS a47bd8425cf4447b825521669499b85e (127.13.223.66:34803): tablet 996a14dfc7cc4de1b672753d3d023d3e (table test-table [id=b6a0b7bea6754e57a7b7cbd72f9afe56]) successfully deleted
I20260928 14:50:09.550967 31791 catalog_manager.cc:4841] TS 315d61f8fbc1410a8dd3587513c932ff (127.13.223.67:46069): tablet 996a14dfc7cc4de1b672753d3d023d3e (table test-table [id=b6a0b7bea6754e57a7b7cbd72f9afe56]) successfully deleted
W20260928 14:50:09.551829 31790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:39889 (error 108)
I20260928 14:50:09.552026 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31997
I20260928 14:50:09.553573 32269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:09.553088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:39889 (local address 127.0.0.1:57668)
0928 14:50:09.553366 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:09.553477 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:39889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260928 14:50:09.553758 31790 catalog_manager.cc:4584] TS 218947a925e349079a05a08cc47ae176 (127.13.223.65:39889): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 996a14dfc7cc4de1b672753d3d023d3e: Network error: Client connection negotiation failed: client connection to 127.13.223.65:39889: connect: Connection refused (error 111)
I20260928 14:50:09.558840 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 32128
I20260928 14:50:09.565102 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 31775
2026-09-28T14:50:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1180 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:09 dist-test-slave-gkhx krb5kdc[32303](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:09 dist-test-slave-gkhx krb5kdc[32303](info): set up 2 sockets
Sep 28 14:50:09 dist-test-slave-gkhx krb5kdc[32303](info): commencing operation
krb5kdc: starting...
W20260928 14:50:10.649204 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:10 dist-test-slave-gkhx krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:10.790671 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:39017
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:39017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:10.907590 32316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:10.907950 32316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:10.908036 32316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:10.912475 32316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:10.912619 32316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:10.912703 32316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:10.912745 32316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:10.912798 32316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:10.918758 32316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:39017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:39017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:10.920014 32316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:10.921167 32316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:50:10.931270 32316 server_base.cc:1034] running on GCE node
W20260928 14:50:10.931223 32321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.931221 32322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:10.931245 32324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:10.932242 32316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:10.933506 32316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:10.934692 32316 hybrid_clock.cc:648] HybridClock initialized: now 1790607010934674 us; error 39 us; skew 500 ppm
Sep 28 14:50:10 dist-test-slave-gkhx krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:10.943181 32316 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:10.944305 32316 webserver.cc:460] Webserver started at http://127.13.223.126:43535/ using document root <none> and password file <none>
I20260928 14:50:10.944932 32316 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:10.944999 32316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:10.945233 32316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:10.947535 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "d2ce271f4a5747518a928e15c39ea019"
format_stamp: "Formatted at 2026-09-28 14:50:10 on dist-test-slave-gkhx"
server_key: "a4fe4c1e1128d877c1872a460473c7c9"
server_key_iv: "a7c08bd2201d5d1980a0159bebcabc87"
server_key_version: "encryptionkey@0"
I20260928 14:50:10.948174 32316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "d2ce271f4a5747518a928e15c39ea019"
format_stamp: "Formatted at 2026-09-28 14:50:10 on dist-test-slave-gkhx"
server_key: "a4fe4c1e1128d877c1872a460473c7c9"
server_key_iv: "a7c08bd2201d5d1980a0159bebcabc87"
server_key_version: "encryptionkey@0"
I20260928 14:50:10.952559 32316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:10.955430 32331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:10.957175 32316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:10.957363 32316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "d2ce271f4a5747518a928e15c39ea019"
format_stamp: "Formatted at 2026-09-28 14:50:10 on dist-test-slave-gkhx"
server_key: "a4fe4c1e1128d877c1872a460473c7c9"
server_key_iv: "a7c08bd2201d5d1980a0159bebcabc87"
server_key_version: "encryptionkey@0"
I20260928 14:50:10.957512 32316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:10.991778 32316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:10.996269 32316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:10.996529 32316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:11.005234 32316 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:39017
I20260928 14:50:11.005299 32383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:39017 every 8 connection(s)
I20260928 14:50:11.006379 32316 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:11.008888 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 32316
I20260928 14:50:11.009063 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:11.009403 14205 external_mini_cluster.cc:1428] Setting key 8ed466343b02f25debad006c2e59ede3
I20260928 14:50:11.009806 32384 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.017831 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: Bootstrap starting.
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:11.021037 32384 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.022241 32384 log.cc:824] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.025388 32384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: No bootstrap required, opened a new log
I20260928 14:50:11.028053 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.011120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34214 (local address 127.13.223.126:39017)
0928 14:50:11.011624 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.011646 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.011691 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:50:11.012478 (+ 787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.012519 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.013016 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.013443 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.014295 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.015960 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.017021 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.017761 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.017948 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.021084 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.021092 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.021117 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.021164 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.023611 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.024293 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.024300 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.024307 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.024498 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.024862 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.024865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.024867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.025524 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.025745 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.026113 (+ 368us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.026480 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":148,"threads_started":1}
I20260928 14:50:11.030062 32384 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } }
I20260928 14:50:11.030328 32384 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.030368 32384 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2ce271f4a5747518a928e15c39ea019, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.030916 32384 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } }
I20260928 14:50:11.031070 32384 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:11.031176 32384 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:11.031325 32384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:11.032416 32384 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } }
I20260928 14:50:11.032850 32384 leader_election.cc:302] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2ce271f4a5747518a928e15c39ea019; no voters:
I20260928 14:50:11.033262 32384 leader_election.cc:288] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:11.033442 32389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:11.033730 32389 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [term 1 LEADER]: Becoming Leader. State: Replica: d2ce271f4a5747518a928e15c39ea019, State: Running, Role: LEADER
I20260928 14:50:11.034118 32389 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } }
I20260928 14:50:11.034809 32384 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:11.036185 32390 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2ce271f4a5747518a928e15c39ea019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } } }
I20260928 14:50:11.036240 32391 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2ce271f4a5747518a928e15c39ea019. Latest consensus state: current_term: 1 leader_uuid: "d2ce271f4a5747518a928e15c39ea019" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2ce271f4a5747518a928e15c39ea019" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 39017 } } }
I20260928 14:50:11.036329 32390 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:11.036355 32391 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:11.037083 32398 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:11.040561 32398 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:11.047885 32398 catalog_manager.cc:1347] Generated new cluster ID: 248577ef9c644cfbbbd7bd303f6da5af
I20260928 14:50:11.048017 32398 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:11.083782 32398 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:11.084880 32398 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:11.097499 32398 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: Generated new TSK 0
I20260928 14:50:11.098879 32398 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:50:11.140174 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:11.271225 32410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:11.271571 32410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:11.271622 32410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:11.276127 32410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:11.276224 32410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:11.276292 32410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:11.282394 32410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:11.283622 32410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:11.284682 32410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:11.295383 32415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:11.295358 32418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:11.295413 32416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:11.295795 32410 server_base.cc:1034] running on GCE node
I20260928 14:50:11.296242 32410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:11.296965 32410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:11.298182 32410 hybrid_clock.cc:648] HybridClock initialized: now 1790607011298146 us; error 61 us; skew 500 ppm
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:11.301263 32410 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:50:11.302263 32410 webserver.cc:460] Webserver started at http://127.13.223.65:43533/ using document root <none> and password file <none>
I20260928 14:50:11.302927 32410 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:11.303037 32410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:11.303270 32410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:11.305429 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "cc4618debb6c45be8d57409404b07910"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "2e8b980495cfb7b40ac46a11f43694fd"
server_key_iv: "1b37d2285e420d0b32c023d9aeba820e"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.306037 32410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "cc4618debb6c45be8d57409404b07910"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "2e8b980495cfb7b40ac46a11f43694fd"
server_key_iv: "1b37d2285e420d0b32c023d9aeba820e"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.310673 32410 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260928 14:50:11.313632 32425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.315448 32410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:11.315622 32410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "cc4618debb6c45be8d57409404b07910"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "2e8b980495cfb7b40ac46a11f43694fd"
server_key_iv: "1b37d2285e420d0b32c023d9aeba820e"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.315783 32410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:11.331707 32410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:11.335748 32410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:11.335986 32410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:11.336715 32410 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:11.337599 32410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:11.337672 32410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.337741 32410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:11.337790 32410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.348242 32410 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:38073
I20260928 14:50:11.348268 32538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:38073 every 8 connection(s)
I20260928 14:50:11.349248 32410 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:11.358865 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 32410
I20260928 14:50:11.358989 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:11.359416 14205 external_mini_cluster.cc:1428] Setting key 04a1b22ebfe59d9e20ee403bde1cbed7
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:11.364987 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.351289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:58511 (local address 127.13.223.126:39017)
0928 14:50:11.351470 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.351473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.352578 (+ 1105us) server_negotiation.cc:407] Connection header received
0928 14:50:11.353937 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.353940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.354005 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.354121 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.355486 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.356048 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.357325 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.357578 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.357696 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.360794 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.360799 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.360802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.360831 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.362522 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.363141 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.363147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.363149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.363246 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.363630 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.363635 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.363636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.363912 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.364020 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.364587 (+ 567us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.364742 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260928 14:50:11.365818 32541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.351739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:39017 (local address 127.13.223.65:58511)
0928 14:50:11.352312 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.352373 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.353705 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.354313 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.354320 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.354778 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:11.355323 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.355338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.356186 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.356190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.357190 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.357200 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.357749 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.357753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.357942 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.358643 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:11.358660 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:11.360568 (+ 1908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:11.362730 (+ 2162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.362736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.362751 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.363038 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.363347 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.363349 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.363350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.363500 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.364032 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:11.364041 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:11.364327 (+ 286us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.364498 (+ 171us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.364825 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":114,"threads_started":1}
I20260928 14:50:11.367409 32539 heartbeater.cc:347] Connected to a master server at 127.13.223.126:39017
I20260928 14:50:11.367744 32539 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.368278 32539 heartbeater.cc:511] Master 127.13.223.126:39017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:50:11.370110 32348 ts_manager.cc:194] Registered new tserver with Master: cc4618debb6c45be8d57409404b07910 (127.13.223.65:38073)
I20260928 14:50:11.371417 32348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:58511
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:50:11.391474 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:11.511318 32544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:11.511667 32544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:11.511751 32544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:11.516017 32544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:11.516119 32544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:11.516222 32544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:11.522183 32544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:11.523443 32544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:11.524547 32544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:11.534951 32550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:11.534951 32549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:11.535283 32544 server_base.cc:1034] running on GCE node
W20260928 14:50:11.535182 32552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:11.535948 32544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:11.536835 32544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:11.538033 32544 hybrid_clock.cc:648] HybridClock initialized: now 1790607011538021 us; error 52 us; skew 500 ppm
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:11.541414 32544 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:50:11.542486 32544 webserver.cc:460] Webserver started at http://127.13.223.66:42125/ using document root <none> and password file <none>
I20260928 14:50:11.543162 32544 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:11.543267 32544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:11.543520 32544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:11.545781 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "6d3afa1f59934204a5320f732f06d144"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "f4404238724086339598a7db9aff1797"
server_key_iv: "97af2031d2a08f19b0f604ba73cb51d9"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.546427 32544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "6d3afa1f59934204a5320f732f06d144"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "f4404238724086339598a7db9aff1797"
server_key_iv: "97af2031d2a08f19b0f604ba73cb51d9"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.550702 32544 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:11.553809 32559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.555384 32544 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:11.555563 32544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "6d3afa1f59934204a5320f732f06d144"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "f4404238724086339598a7db9aff1797"
server_key_iv: "97af2031d2a08f19b0f604ba73cb51d9"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.555747 32544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:11.570253 32544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:11.574043 32544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:11.574339 32544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:11.575162 32544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:11.576201 32544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:11.576283 32544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.576362 32544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:11.576426 32544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.587795 32544 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:46143
I20260928 14:50:11.587795 32672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:46143 every 8 connection(s)
I20260928 14:50:11.588848 32544 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:11.589285 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 32544
I20260928 14:50:11.589437 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:11.589794 14205 external_mini_cluster.cc:1428] Setting key de6a6812586aac19bfb28df1b0d53dbd
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:50:11.605844 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.591215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:41055 (local address 127.13.223.126:39017)
0928 14:50:11.591390 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.591395 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.592712 (+ 1317us) server_negotiation.cc:407] Connection header received
0928 14:50:11.594184 (+ 1472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.594188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.594249 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.594336 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.595681 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.596242 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.597559 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.597803 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.597922 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.601326 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.601330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.601333 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.601371 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.603072 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.603835 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.603839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.603840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.603941 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.604349 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.604353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.604354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.604614 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.604734 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.605533 (+ 799us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.605699 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:50:11.607031 32675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.591917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:39017 (local address 127.13.223.66:41055)
0928 14:50:11.592544 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.592592 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.593951 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.594524 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.594531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.594993 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:11.595519 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.595534 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.596386 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.596392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.597414 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.597425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.597984 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.597989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.598180 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.598927 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:11.598951 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:11.601104 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:11.603258 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.603269 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.603294 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.603703 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.604053 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.604056 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.604058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.604241 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.604736 (+ 495us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:11.604747 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:11.605201 (+ 454us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.605446 (+ 245us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.605874 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":161,"threads_started":1}
I20260928 14:50:11.608309 32673 heartbeater.cc:347] Connected to a master server at 127.13.223.126:39017
I20260928 14:50:11.608598 32673 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.609257 32673 heartbeater.cc:511] Master 127.13.223.126:39017 requested a full tablet report, sending...
I20260928 14:50:11.610453 32348 ts_manager.cc:194] Registered new tserver with Master: 6d3afa1f59934204a5320f732f06d144 (127.13.223.66:46143)
I20260928 14:50:11.611110 32348 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:41055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:50:11.625882 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:45667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:11.745210 32678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:11.745520 32678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:11.745566 32678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:11.749960 32678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:11.750048 32678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:11.750113 32678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:11.755841 32678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:39017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:11.756946 32678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:11.757977 32678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:50:11.769402 32678 server_base.cc:1034] running on GCE node
W20260928 14:50:11.769354 32684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:11.769354 32683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:11.769351 32686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:11.770264 32678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:11.771157 32678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:11.772346 32678 hybrid_clock.cc:648] HybridClock initialized: now 1790607011772282 us; error 82 us; skew 500 ppm
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:11.775588 32678 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:50:11.776755 32678 webserver.cc:460] Webserver started at http://127.13.223.67:37817/ using document root <none> and password file <none>
I20260928 14:50:11.777420 32678 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:11.777520 32678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:11.777804 32678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:11.780208 32678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "4972f33c51a946b9b7951aca314633c7"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "52370a2ee8c8d1d59888f2499a9569ba"
server_key_iv: "114df4218c21f6f033be96113a4bcaec"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.780861 32678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "4972f33c51a946b9b7951aca314633c7"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "52370a2ee8c8d1d59888f2499a9569ba"
server_key_iv: "114df4218c21f6f033be96113a4bcaec"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.785465 32678 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:11.788807 32693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.790755 32678 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:11.790933 32678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "4972f33c51a946b9b7951aca314633c7"
format_stamp: "Formatted at 2026-09-28 14:50:11 on dist-test-slave-gkhx"
server_key: "52370a2ee8c8d1d59888f2499a9569ba"
server_key_iv: "114df4218c21f6f033be96113a4bcaec"
server_key_version: "encryptionkey@0"
I20260928 14:50:11.791097 32678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:11.828333 32678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:11.832811 32678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:11.833081 32678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:11.833827 32678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:11.835016 32678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:11.835104 32678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.835213 32678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:11.835268 32678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:11.846963 32678 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:46423
I20260928 14:50:11.847067 339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:46423 every 8 connection(s)
I20260928 14:50:11.848094 32678 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:11.854522 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 32678
I20260928 14:50:11.854737 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:11.855052 14205 external_mini_cluster.cc:1428] Setting key 781d2004c2e2fbffb2a2d863b0bf4390
Sep 28 14:50:11 dist-test-slave-gkhx krb5kdc[32303](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607011, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:11.864796 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.850383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:51747 (local address 127.13.223.126:39017)
0928 14:50:11.850579 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.850584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.851550 (+ 966us) server_negotiation.cc:407] Connection header received
0928 14:50:11.852975 (+ 1425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.852978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.853034 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.853126 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.854438 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.855228 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.856880 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.857213 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.857325 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.860367 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.860372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.860374 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.860404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.862261 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.862992 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.862996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.862997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.863095 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.863529 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.863533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.863534 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.863752 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.863839 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.864527 (+ 688us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.864653 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:50:11.865629 342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.850842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:39017 (local address 127.13.223.67:51747)
0928 14:50:11.851371 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.851422 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.852715 (+ 1293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.853288 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.853295 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.853683 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:11.854262 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.854285 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.855365 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.855371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.856761 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.856770 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.857388 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.857391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.857558 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.858197 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:11.858215 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:11.860131 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:11.862458 (+ 2327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.862464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.862480 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.862858 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.863232 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:11.863235 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:11.863239 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:11.863401 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:11.863862 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:11.863868 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:11.864207 (+ 339us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.864377 (+ 170us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.864681 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":125,"threads_started":1}
I20260928 14:50:11.867053 340 heartbeater.cc:347] Connected to a master server at 127.13.223.126:39017
I20260928 14:50:11.867316 340 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:11.867965 340 heartbeater.cc:511] Master 127.13.223.126:39017 requested a full tablet report, sending...
I20260928 14:50:11.869333 32347 ts_manager.cc:194] Registered new tserver with Master: 4972f33c51a946b9b7951aca314633c7 (127.13.223.67:46423)
I20260928 14:50:11.870045 32347 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:51747
I20260928 14:50:11.881839 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:11.893553 32387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.884288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34228 (local address 127.13.223.126:39017)
0928 14:50:11.884479 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.884483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.884577 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:50:11.884692 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.884695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.884737 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.884807 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:11.885572 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.886173 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.887374 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.887605 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.887736 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.889434 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:11.889437 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:11.889439 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:11.889468 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:11.891179 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.891823 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.891831 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.891834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.891948 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:11.892383 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:11.892386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:11.892388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:11.892654 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:11.892802 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.893126 (+ 324us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.893272 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260928 14:50:11.897351 32347 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:11.900892 32347 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:11.917173 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.911746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.13.223.65:38073)
0928 14:50:11.912216 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.912223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.912245 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:50:11.912322 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.912327 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.912528 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.912796 (+ 268us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.913085 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.913825 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.915354 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.916282 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.916432 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.916606 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.916853 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.916992 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":76,"threads_started":1}
I20260928 14:50:11.917172 355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.911311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:38073 (local address 127.0.0.1:51062)
0928 14:50:11.911640 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.911653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.911736 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.912673 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.912675 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.912693 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.912956 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.912961 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.913943 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.913949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.915216 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.915226 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.916470 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.916474 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.916608 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.916659 (+ 51us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.916805 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.916951 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":146,"thread_start_us":95,"threads_started":1}
I20260928 14:50:11.917516 352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.910459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:46143 (local address 127.0.0.1:48108)
0928 14:50:11.910909 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.910962 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.911103 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.912606 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.912609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.912631 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.912884 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.912896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.914641 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.914646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.915930 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.915944 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.917079 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.917082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.917195 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.917227 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.917286 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.917382 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":140,"thread_start_us":193,"threads_started":1}
I20260928 14:50:11.917965 354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.910914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:46423 (local address 127.0.0.1:40228)
0928 14:50:11.911354 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:11.911369 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:11.911455 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:11.912514 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:11.912521 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:11.912552 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:11.912759 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.912767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.914641 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.914646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.916201 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:11.916210 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.917458 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.917463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:11.917576 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.917603 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 14:50:11.917712 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 14:50:11.917787 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"mutex_wait_us":343,"thread_start_us":102,"threads_started":1}
I20260928 14:50:11.918151 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.910629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48108 (local address 127.13.223.66:46143)
0928 14:50:11.911894 (+ 1265us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.911921 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.911953 (+ 32us) server_negotiation.cc:407] Connection header received
0928 14:50:11.912060 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.912066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.912420 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.912818 (+ 398us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.913022 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.914420 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.916081 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.916928 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.917689 (+ 761us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.917858 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.917910 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.918006 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1124,"thread_start_us":166,"threads_started":1}
I20260928 14:50:11.918917 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:11.911434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.13.223.67:46423)
0928 14:50:11.911951 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:11.911965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:11.911988 (+ 23us) server_negotiation.cc:407] Connection header received
0928 14:50:11.912064 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:11.912069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:11.912311 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:11.912527 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:11.913044 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.914391 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.916339 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:11.917329 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:11.917463 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:11.917626 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:11.918603 (+ 977us) server_negotiation.cc:299] Negotiation successful
0928 14:50:11.918727 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":194,"threads_started":1}
I20260928 14:50:11.920929 32473 tablet_service.cc:1467] Processing CreateTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d (DEFAULT_TABLE table=test-table [id=13cf6f8d58444df990c3f9b1d1221a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.921404 32607 tablet_service.cc:1467] Processing CreateTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d (DEFAULT_TABLE table=test-table [id=13cf6f8d58444df990c3f9b1d1221a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.922036 32473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810582ebbd9a45eea6bc0f0418d47b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.922021 32742 tablet_service.cc:1467] Processing CreateTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d (DEFAULT_TABLE table=test-table [id=13cf6f8d58444df990c3f9b1d1221a04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:11.922628 32607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810582ebbd9a45eea6bc0f0418d47b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.923138 32742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 810582ebbd9a45eea6bc0f0418d47b8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:11.929149 358 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Bootstrap starting.
I20260928 14:50:11.931151 359 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Bootstrap starting.
I20260928 14:50:11.932468 358 tablet_bootstrap.cc:654] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.932716 360 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Bootstrap starting.
I20260928 14:50:11.933678 358 log.cc:824] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.933774 359 tablet_bootstrap.cc:654] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.935161 360 tablet_bootstrap.cc:654] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:11.935400 359 log.cc:824] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.936616 358 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: No bootstrap required, opened a new log
I20260928 14:50:11.936709 360 log.cc:824] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:11.936851 358 ts_tablet_manager.cc:1397] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260928 14:50:11.937983 359 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: No bootstrap required, opened a new log
I20260928 14:50:11.938226 359 ts_tablet_manager.cc:1397] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:50:11.939240 360 tablet_bootstrap.cc:492] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: No bootstrap required, opened a new log
I20260928 14:50:11.939459 360 ts_tablet_manager.cc:1397] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:50:11.947449 358 raft_consensus.cc:348] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.947985 358 raft_consensus.cc:374] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.948069 358 raft_consensus.cc:729] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc4618debb6c45be8d57409404b07910, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.948794 358 consensus_queue.cc:260] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.951088 360 raft_consensus.cc:348] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.951488 360 raft_consensus.cc:374] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.951562 358 ts_tablet_manager.cc:1428] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Time spent starting tablet: real 0.015s user 0.009s sys 0.000s
I20260928 14:50:11.951574 360 raft_consensus.cc:729] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4972f33c51a946b9b7951aca314633c7, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.952075 32539 heartbeater.cc:503] Master 127.13.223.126:39017 was elected leader, sending a full tablet report...
I20260928 14:50:11.952481 359 raft_consensus.cc:348] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.952776 359 raft_consensus.cc:374] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:11.952869 359 raft_consensus.cc:729] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d3afa1f59934204a5320f732f06d144, State: Initialized, Role: FOLLOWER
I20260928 14:50:11.953437 359 consensus_queue.cc:260] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.953615 360 consensus_queue.cc:260] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:11.955029 359 ts_tablet_manager.cc:1428] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260928 14:50:11.955453 32673 heartbeater.cc:503] Master 127.13.223.126:39017 was elected leader, sending a full tablet report...
I20260928 14:50:11.955727 360 ts_tablet_manager.cc:1428] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260928 14:50:11.956256 340 heartbeater.cc:503] Master 127.13.223.126:39017 was elected leader, sending a full tablet report...
W20260928 14:50:12.090510 32674 tablet.cc:2369] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:12.099304 341 tablet.cc:2369] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:12.100876 32540 tablet.cc:2369] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:12.215885 364 raft_consensus.cc:482] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:12.216264 364 raft_consensus.cc:504] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:12.217485 364 leader_election.cc:288] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d3afa1f59934204a5320f732f06d144 (127.13.223.66:46143), 4972f33c51a946b9b7951aca314633c7 (127.13.223.67:46423)
I20260928 14:50:12.222594 353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.217957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:41329 (local address 127.13.223.66:46143)
0928 14:50:12.218097 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:12.218102 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:12.218402 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:50:12.218609 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:12.218617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:12.218662 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:12.218744 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:12.219101 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.219947 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.221357 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.221993 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.222135 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.222188 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:12.222356 (+ 168us) server_negotiation.cc:299] Negotiation successful
0928 14:50:12.222438 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:50:12.222704 356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.218149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:41145 (local address 127.13.223.67:46423)
0928 14:50:12.218340 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:12.218344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:12.218494 (+ 150us) server_negotiation.cc:407] Connection header received
0928 14:50:12.218665 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:12.218668 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:12.218724 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:12.218813 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:12.219272 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.219999 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.221404 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.222012 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.222134 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.222188 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:12.222484 (+ 296us) server_negotiation.cc:299] Negotiation successful
0928 14:50:12.222594 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 14:50:12.222648 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.217860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:46143 (local address 127.13.223.65:41329)
0928 14:50:12.218276 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:12.218292 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:12.218450 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:12.218766 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:12.218769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:12.218785 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:12.218961 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.218969 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.220072 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.220075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.221225 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.221234 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.222132 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.222135 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.222212 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.222234 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:50:12.222333 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:50:12.222413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":93,"spinlock_wait_cycles":15616,"thread_start_us":129,"threads_started":1}
I20260928 14:50:12.222723 368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.217860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:46423 (local address 127.13.223.65:41145)
0928 14:50:12.218388 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:12.218407 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:12.218536 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:12.218821 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:12.218824 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:12.218883 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:12.219004 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.219009 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.220104 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.220108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.221222 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.221233 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.222265 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.222268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.222355 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.222373 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 14:50:12.222466 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 14:50:12.222540 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"mutex_wait_us":54,"spinlock_wait_cycles":896,"thread_start_us":95,"threads_started":1}
I20260928 14:50:12.223632 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810582ebbd9a45eea6bc0f0418d47b8d" candidate_uuid: "cc4618debb6c45be8d57409404b07910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3afa1f59934204a5320f732f06d144" is_pre_election: true
I20260928 14:50:12.223717 32762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810582ebbd9a45eea6bc0f0418d47b8d" candidate_uuid: "cc4618debb6c45be8d57409404b07910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4972f33c51a946b9b7951aca314633c7" is_pre_election: true
I20260928 14:50:12.223965 32627 raft_consensus.cc:2454] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4618debb6c45be8d57409404b07910 in term 0.
I20260928 14:50:12.224121 32762 raft_consensus.cc:2454] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc4618debb6c45be8d57409404b07910 in term 0.
I20260928 14:50:12.224743 32428 leader_election.cc:302] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4972f33c51a946b9b7951aca314633c7, 6d3afa1f59934204a5320f732f06d144, cc4618debb6c45be8d57409404b07910; no voters:
I20260928 14:50:12.225097 364 raft_consensus.cc:2781] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:12.225198 364 raft_consensus.cc:482] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:12.225255 364 raft_consensus.cc:3037] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:12.226540 364 raft_consensus.cc:504] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:12.227007 364 leader_election.cc:288] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [CANDIDATE]: Term 1 election: Requested vote from peers 6d3afa1f59934204a5320f732f06d144 (127.13.223.66:46143), 4972f33c51a946b9b7951aca314633c7 (127.13.223.67:46423)
I20260928 14:50:12.227514 32627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810582ebbd9a45eea6bc0f0418d47b8d" candidate_uuid: "cc4618debb6c45be8d57409404b07910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d3afa1f59934204a5320f732f06d144"
I20260928 14:50:12.227514 32762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "810582ebbd9a45eea6bc0f0418d47b8d" candidate_uuid: "cc4618debb6c45be8d57409404b07910" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4972f33c51a946b9b7951aca314633c7"
I20260928 14:50:12.227679 32627 raft_consensus.cc:3037] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:12.227679 32762 raft_consensus.cc:3037] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:12.229218 32762 raft_consensus.cc:2454] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4618debb6c45be8d57409404b07910 in term 1.
I20260928 14:50:12.229218 32627 raft_consensus.cc:2454] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc4618debb6c45be8d57409404b07910 in term 1.
I20260928 14:50:12.229687 32427 leader_election.cc:302] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4972f33c51a946b9b7951aca314633c7, cc4618debb6c45be8d57409404b07910; no voters:
I20260928 14:50:12.229907 364 raft_consensus.cc:2781] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:12.230206 364 raft_consensus.cc:686] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 1 LEADER]: Becoming Leader. State: Replica: cc4618debb6c45be8d57409404b07910, State: Running, Role: LEADER
I20260928 14:50:12.230659 364 consensus_queue.cc:237] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } }
I20260928 14:50:12.234216 32348 catalog_manager.cc:5495] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc4618debb6c45be8d57409404b07910 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "cc4618debb6c45be8d57409404b07910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc4618debb6c45be8d57409404b07910" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38073 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:12.300868 357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.296747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.13.223.65:38073)
0928 14:50:12.296887 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:12.296892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:12.296907 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:50:12.297092 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:12.297094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:12.297150 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:12.297242 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:12.297546 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.298095 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.299211 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.299452 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.299555 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.299893 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:12.300348 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:12.300464 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:12.300704 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:50:12.300749 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:50:12.308249 32627 raft_consensus.cc:1261] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 1 FOLLOWER]: Refusing update from remote peer cc4618debb6c45be8d57409404b07910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:12.308259 32762 raft_consensus.cc:1261] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 1 FOLLOWER]: Refusing update from remote peer cc4618debb6c45be8d57409404b07910: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:12.309495 364 consensus_queue.cc:1035] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4972f33c51a946b9b7951aca314633c7" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46423 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:12.309793 369 consensus_queue.cc:1035] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d3afa1f59934204a5320f732f06d144" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:12.330869 32348 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:12.331132 32348 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:12.334494 32348 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d2ce271f4a5747518a928e15c39ea019: Sending DeleteTablet for 3 replicas of tablet 810582ebbd9a45eea6bc0f0418d47b8d
I20260928 14:50:12.335703 32742 tablet_service.cc:1514] Processing DeleteTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:12 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:40228
I20260928 14:50:12.335703 32607 tablet_service.cc:1514] Processing DeleteTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:12 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:48108
I20260928 14:50:12.336517 383 tablet_replica.cc:330] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: stopping tablet replica
I20260928 14:50:12.336548 32473 tablet_service.cc:1514] Processing DeleteTablet for tablet 810582ebbd9a45eea6bc0f0418d47b8d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:12 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:51062
I20260928 14:50:12.336952 383 raft_consensus.cc:2229] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:12.337180 384 tablet_replica.cc:330] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910: stopping tablet replica
I20260928 14:50:12.337423 383 raft_consensus.cc:2258] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:12.337522 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 32410
I20260928 14:50:12.337716 384 raft_consensus.cc:2229] T 810582ebbd9a45eea6bc0f0418d47b8d P cc4618debb6c45be8d57409404b07910 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:12.338963 382 tablet_replica.cc:330] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: stopping tablet replica
I20260928 14:50:12.339434 382 raft_consensus.cc:2229] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:12.339615 383 ts_tablet_manager.cc:1905] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:12.339939 382 raft_consensus.cc:2258] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:12.341830 382 ts_tablet_manager.cc:1905] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:12.342453 383 ts_tablet_manager.cc:1918] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:12.342608 383 log.cc:1196] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/810582ebbd9a45eea6bc0f0418d47b8d
I20260928 14:50:12.342913 383 ts_tablet_manager.cc:1939] T 810582ebbd9a45eea6bc0f0418d47b8d P 4972f33c51a946b9b7951aca314633c7: Deleting consensus metadata
I20260928 14:50:12.344374 32333 catalog_manager.cc:4841] TS 4972f33c51a946b9b7951aca314633c7 (127.13.223.67:46423): tablet 810582ebbd9a45eea6bc0f0418d47b8d (table test-table [id=13cf6f8d58444df990c3f9b1d1221a04]) successfully deleted
I20260928 14:50:12.345397 382 ts_tablet_manager.cc:1918] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:12.345521 382 log.cc:1196] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/810582ebbd9a45eea6bc0f0418d47b8d
I20260928 14:50:12.345860 382 ts_tablet_manager.cc:1939] T 810582ebbd9a45eea6bc0f0418d47b8d P 6d3afa1f59934204a5320f732f06d144: Deleting consensus metadata
W20260928 14:50:12.347440 32334 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:38073 (error 108)
I20260928 14:50:12.348024 32334 catalog_manager.cc:4841] TS 6d3afa1f59934204a5320f732f06d144 (127.13.223.66:46143): tablet 810582ebbd9a45eea6bc0f0418d47b8d (table test-table [id=13cf6f8d58444df990c3f9b1d1221a04]) successfully deleted
W20260928 14:50:12.348742 354 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:12.348365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:38073 (local address 127.0.0.1:51082)
0928 14:50:12.348530 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:12.348614 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:38073: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 14:50:12.348881 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 32544
W20260928 14:50:12.348934 32334 catalog_manager.cc:4584] TS cc4618debb6c45be8d57409404b07910 (127.13.223.65:38073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 810582ebbd9a45eea6bc0f0418d47b8d: Network error: Client connection negotiation failed: client connection to 127.13.223.65:38073: connect: Connection reset by peer (error 104)
I20260928 14:50:12.355341 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 32678
I20260928 14:50:12.361472 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 32316
2026-09-28T14:50:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2797 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T14:50:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:12Z Disabled control of system clock
I20260928 14:50:12.398255 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37663
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:12.518944 389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:12.519271 389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:12.519309 389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:12.523700 389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:12.523801 389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:12.523818 389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:12.523856 389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:12.523873 389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:12.529253 389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:37663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:12.530633 389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:12.532141 389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:12.541297 398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:12.541297 395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:12.541693 396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:12.541677 389 server_base.cc:1034] running on GCE node
I20260928 14:50:12.542379 389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:12.543650 389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:12.544834 389 hybrid_clock.cc:648] HybridClock initialized: now 1790607012544790 us; error 55 us; skew 500 ppm
I20260928 14:50:12.546972 389 webserver.cc:460] Webserver started at http://127.13.223.126:44947/ using document root <none> and password file <none>
I20260928 14:50:12.547600 389 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:12.547667 389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:12.547899 389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:12.550084 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "4445496264ed42ac8febebce2ae45882"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "2060a32e11bf5c618454e1a74b2dc487"
server_key_iv: "e3ba054fcb3088728e8c6b9fef681652"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.550742 389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "4445496264ed42ac8febebce2ae45882"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "2060a32e11bf5c618454e1a74b2dc487"
server_key_iv: "e3ba054fcb3088728e8c6b9fef681652"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.554988 389 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:12.557778 404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:12.559478 389 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:12.559672 389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "4445496264ed42ac8febebce2ae45882"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "2060a32e11bf5c618454e1a74b2dc487"
server_key_iv: "e3ba054fcb3088728e8c6b9fef681652"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.559834 389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:12.574817 389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:12.575695 389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:12.575953 389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:12.584664 389 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:37663
I20260928 14:50:12.584699 458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:37663 every 8 connection(s)
I20260928 14:50:12.585810 389 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:12.586092 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 389
I20260928 14:50:12.586237 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:12.586659 14205 external_mini_cluster.cc:1428] Setting key 0a4a89043b95764bae7ecb8d6107eead
I20260928 14:50:12.589896 459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:12.596232 459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: Bootstrap starting.
I20260928 14:50:12.598076 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.588297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43842 (local address 127.13.223.126:37663)
0928 14:50:12.589180 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:12.589206 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:12.589251 (+ 45us) server_negotiation.cc:407] Connection header received
0928 14:50:12.590011 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:12.590076 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:12.590656 (+ 580us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:12.591066 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:12.591577 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.592714 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.594194 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.594839 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.595028 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.595976 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:12.595981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:12.595997 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:12.596444 (+ 447us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:12.596526 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:12.596555 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:12.596663 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:12.596799 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 14:50:12.597306 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":216,"threads_started":1}
I20260928 14:50:12.598822 459 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:12.599648 459 log.cc:824] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:12.602403 459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: No bootstrap required, opened a new log
I20260928 14:50:12.606889 459 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } }
I20260928 14:50:12.607239 459 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:12.607295 459 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4445496264ed42ac8febebce2ae45882, State: Initialized, Role: FOLLOWER
I20260928 14:50:12.607805 459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } }
I20260928 14:50:12.607999 459 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:12.608080 459 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:12.608229 459 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:12.609364 459 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } }
I20260928 14:50:12.609723 459 leader_election.cc:302] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4445496264ed42ac8febebce2ae45882; no voters:
I20260928 14:50:12.610041 459 leader_election.cc:288] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:12.610316 464 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:12.610579 464 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [term 1 LEADER]: Becoming Leader. State: Replica: 4445496264ed42ac8febebce2ae45882, State: Running, Role: LEADER
I20260928 14:50:12.610917 464 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } }
I20260928 14:50:12.611493 459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:12.612983 465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4445496264ed42ac8febebce2ae45882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } } }
I20260928 14:50:12.613122 465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:12.613039 466 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4445496264ed42ac8febebce2ae45882. Latest consensus state: current_term: 1 leader_uuid: "4445496264ed42ac8febebce2ae45882" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4445496264ed42ac8febebce2ae45882" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 37663 } } }
I20260928 14:50:12.613191 466 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:12.613983 473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:12.616993 473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:12.622865 473 catalog_manager.cc:1347] Generated new cluster ID: 600d9090ff804e4eb070803b745feceb
I20260928 14:50:12.622954 473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:12.635744 473 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:12.637219 473 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:12.642611 473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: Generated new TSK 0
I20260928 14:50:12.643488 473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:12.655126 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--builtin_ntp_servers=127.13.223.84:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:12.777513 483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:12.777843 483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:12.777915 483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:12.782196 483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:12.782289 483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:12.782384 483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:12.787633 483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:12.788797 483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:12.790215 483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:12.800031 491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:12.800061 488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:12.800127 489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:12.800312 483 server_base.cc:1034] running on GCE node
I20260928 14:50:12.800777 483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:12.801486 483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:12.802636 483 hybrid_clock.cc:648] HybridClock initialized: now 1790607012802611 us; error 38 us; skew 500 ppm
I20260928 14:50:12.804634 483 webserver.cc:460] Webserver started at http://127.13.223.65:38381/ using document root <none> and password file <none>
I20260928 14:50:12.805241 483 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:12.805330 483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:12.805559 483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:12.807751 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "e6fb0a11b4014de7a59868ff901b05b7"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "5140711da0a84ba846a80977c3c3f839"
server_key_iv: "d90bd116c7b0fadde83b54ac752627d1"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.808362 483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "e6fb0a11b4014de7a59868ff901b05b7"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "5140711da0a84ba846a80977c3c3f839"
server_key_iv: "d90bd116c7b0fadde83b54ac752627d1"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.812572 483 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:12.815395 497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:12.816833 483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:50:12.817049 483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "e6fb0a11b4014de7a59868ff901b05b7"
format_stamp: "Formatted at 2026-09-28 14:50:12 on dist-test-slave-gkhx"
server_key: "5140711da0a84ba846a80977c3c3f839"
server_key_iv: "d90bd116c7b0fadde83b54ac752627d1"
server_key_version: "encryptionkey@0"
I20260928 14:50:12.817188 483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:12.834326 483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:12.835165 483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:12.835381 483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:12.836009 483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:12.837004 483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:12.837072 483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:12.837147 483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:12.837188 483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:12.847095 483 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:46001
I20260928 14:50:12.847193 612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:46001 every 8 connection(s)
I20260928 14:50:12.848129 483 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:12.852665 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 483
I20260928 14:50:12.852840 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:12.853272 14205 external_mini_cluster.cc:1428] Setting key 7b6a5b378a8261826c82235de9e9d213
I20260928 14:50:12.855358 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--builtin_ntp_servers=127.13.223.84:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:12.858837 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.850167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:44955 (local address 127.13.223.126:37663)
0928 14:50:12.850318 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:12.850323 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:12.851552 (+ 1229us) server_negotiation.cc:407] Connection header received
0928 14:50:12.853127 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:12.853133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:12.853203 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:12.853313 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:12.854334 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.854915 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.856420 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.856739 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.856868 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.858312 (+ 1444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:12.858315 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:12.858317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:12.858331 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:12.858353 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:12.858361 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:12.858424 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:12.858627 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:50:12.858713 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:50:12.860198 615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:12.850679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37663 (local address 127.13.223.65:44955)
0928 14:50:12.851385 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:12.851431 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:12.852843 (+ 1412us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:12.853528 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:12.853535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:12.853657 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:12.854186 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.854200 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.855076 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.855081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.856270 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:12.856282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:12.856972 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:12.856975 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:12.857204 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:12.857968 (+ 764us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:12.857970 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:12.858031 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:12.858136 (+ 105us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:12.858449 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:12.858462 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:12.858471 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 14:50:12.858621 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 14:50:12.858965 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":119,"threads_started":1}
I20260928 14:50:12.861748 613 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37663
I20260928 14:50:12.862046 613 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:12.862775 613 heartbeater.cc:511] Master 127.13.223.126:37663 requested a full tablet report, sending...
I20260928 14:50:12.864578 419 ts_manager.cc:194] Registered new tserver with Master: e6fb0a11b4014de7a59868ff901b05b7 (127.13.223.65:46001)
I20260928 14:50:12.866060 419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.65:44955
W20260928 14:50:12.985769 616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:12.986213 616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:12.986294 616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:12.991178 616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:12.991302 616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:12.991413 616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:12.997714 616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.616
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:12.999210 616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:13.000905 616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:13.013082 621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:13.013140 616 server_base.cc:1034] running on GCE node
W20260928 14:50:13.013007 624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:13.013034 622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:13.013796 616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:13.014659 616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:13.015868 616 hybrid_clock.cc:648] HybridClock initialized: now 1790607013015846 us; error 36 us; skew 500 ppm
I20260928 14:50:13.018321 616 webserver.cc:460] Webserver started at http://127.13.223.66:39327/ using document root <none> and password file <none>
I20260928 14:50:13.019115 616 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:13.019217 616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:13.019438 616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:13.021905 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "107e019d74bb45bd88cb90f0bce30c66"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "ce721d86c4fbda809735f1a22393d4ca"
server_key_iv: "3d8df5b008a2e53f35b65d51e0825745"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.022526 616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "107e019d74bb45bd88cb90f0bce30c66"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "ce721d86c4fbda809735f1a22393d4ca"
server_key_iv: "3d8df5b008a2e53f35b65d51e0825745"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.026865 616 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:13.029772 630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.031347 616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:13.031513 616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "107e019d74bb45bd88cb90f0bce30c66"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "ce721d86c4fbda809735f1a22393d4ca"
server_key_iv: "3d8df5b008a2e53f35b65d51e0825745"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.031694 616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:13.060177 616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:13.061059 616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:13.061292 616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:13.061945 616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:13.063222 616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:13.063295 616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.063367 616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:13.063413 616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.073211 616 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:35843
I20260928 14:50:13.073236 745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:35843 every 8 connection(s)
I20260928 14:50:13.074293 616 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:13.084156 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 616
I20260928 14:50:13.084309 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:13.084689 14205 external_mini_cluster.cc:1428] Setting key e45837aceed1f0aabd1fdb8809b9fee0
I20260928 14:50:13.084740 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.076445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:59209 (local address 127.13.223.126:37663)
0928 14:50:13.076623 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.076628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.077830 (+ 1202us) server_negotiation.cc:407] Connection header received
0928 14:50:13.079439 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.079442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.079497 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.079582 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:13.080618 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.081182 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.082464 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.082769 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.082854 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.084159 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:13.084163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:13.084166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:13.084185 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:13.084212 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:13.084225 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:13.084314 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.084503 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.084583 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:50:13.085891 748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.076954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37663 (local address 127.13.223.66:59209)
0928 14:50:13.077668 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.077711 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.079202 (+ 1491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.079777 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.079784 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.079876 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:13.080467 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.080481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.081316 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.081319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.082340 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.082351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.082919 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.082921 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.083133 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.083851 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:13.083852 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:13.083899 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:13.083973 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:13.084329 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:13.084339 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:13.084344 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.084472 (+ 128us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.084781 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":133,"threads_started":1}
I20260928 14:50:13.087311 746 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37663
I20260928 14:50:13.087357 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--builtin_ntp_servers=127.13.223.84:45915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:13.087651 746 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:13.088349 746 heartbeater.cc:511] Master 127.13.223.126:37663 requested a full tablet report, sending...
I20260928 14:50:13.089767 419 ts_manager.cc:194] Registered new tserver with Master: 107e019d74bb45bd88cb90f0bce30c66 (127.13.223.66:35843)
I20260928 14:50:13.090966 419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.66:59209
W20260928 14:50:13.215281 749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:13.215593 749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:13.215636 749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:13.219866 749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:13.219938 749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:13.220002 749 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:13.225363 749 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:45915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:37663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.749
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:13.226519 749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:13.228052 749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:13.238847 757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:13.238819 755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:13.238847 754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:13.239115 749 server_base.cc:1034] running on GCE node
I20260928 14:50:13.239719 749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:13.240517 749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:13.241729 749 hybrid_clock.cc:648] HybridClock initialized: now 1790607013241710 us; error 39 us; skew 500 ppm
I20260928 14:50:13.243862 749 webserver.cc:460] Webserver started at http://127.13.223.67:40417/ using document root <none> and password file <none>
I20260928 14:50:13.244534 749 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:13.244634 749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:13.244887 749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:13.247107 749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "48827e1f7a6645859a8dfdc0245bf5aa"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "62eb1d12b6b511aebc2983063f4524a9"
server_key_iv: "57c363b913ed6020732525a0a56946ab"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.247678 749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "48827e1f7a6645859a8dfdc0245bf5aa"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "62eb1d12b6b511aebc2983063f4524a9"
server_key_iv: "57c363b913ed6020732525a0a56946ab"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.251843 749 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:13.254829 763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.256278 749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:50:13.256392 749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "48827e1f7a6645859a8dfdc0245bf5aa"
format_stamp: "Formatted at 2026-09-28 14:50:13 on dist-test-slave-gkhx"
server_key: "62eb1d12b6b511aebc2983063f4524a9"
server_key_iv: "57c363b913ed6020732525a0a56946ab"
server_key_version: "encryptionkey@0"
I20260928 14:50:13.256510 749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:13.293330 749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:13.294016 749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:13.294183 749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:13.294809 749 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:13.295895 749 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:13.295950 749 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.295991 749 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:13.296006 749 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:13.305150 749 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:40387
I20260928 14:50:13.305187 878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:40387 every 8 connection(s)
I20260928 14:50:13.306102 749 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:13.315750 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.307922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:51527 (local address 127.13.223.126:37663)
0928 14:50:13.308118 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.308123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.309229 (+ 1106us) server_negotiation.cc:407] Connection header received
0928 14:50:13.310742 (+ 1513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.310745 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.310792 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.310869 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:13.311868 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.312407 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.313822 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.314041 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.314123 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.315275 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:13.315278 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:13.315280 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:13.315302 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:13.315320 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:13.315328 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:13.315386 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.315542 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.315590 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260928 14:50:13.316025 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 749
I20260928 14:50:13.316143 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:13.316450 14205 external_mini_cluster.cc:1428] Setting key 48c137389c9f3b849603a92c156f0e83
I20260928 14:50:13.316900 881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.308330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:37663 (local address 127.13.223.67:51527)
0928 14:50:13.309049 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.309096 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.310466 (+ 1370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.311041 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.311048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.311135 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:13.311690 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.311704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.312547 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.312550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.313663 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.313674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.314204 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.314207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.314375 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.315014 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:13.315016 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:13.315066 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:13.315134 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:13.315408 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:13.315416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:13.315421 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.315531 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.315814 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":117,"threads_started":1}
I20260928 14:50:13.318140 879 heartbeater.cc:347] Connected to a master server at 127.13.223.126:37663
I20260928 14:50:13.318391 879 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:13.319036 879 heartbeater.cc:511] Master 127.13.223.126:37663 requested a full tablet report, sending...
I20260928 14:50:13.320130 419 ts_manager.cc:194] Registered new tserver with Master: 48827e1f7a6645859a8dfdc0245bf5aa (127.13.223.67:40387)
I20260928 14:50:13.320870 419 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.67:51527
I20260928 14:50:13.330682 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:13.337034 462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.332802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.13.223.126:37663)
0928 14:50:13.333058 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.333065 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.333147 (+ 82us) server_negotiation.cc:407] Connection header received
0928 14:50:13.333305 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.333308 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.333358 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.333442 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:13.333763 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.334299 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.335225 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.335538 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.335645 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.336480 (+ 835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:13.336484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:13.336486 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:13.336500 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:13.336523 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:13.336534 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:13.336694 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.336836 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.336872 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260928 14:50:13.339968 419 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43854:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:13.343072 419 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:13.358202 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.351385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.13.223.66:35843)
0928 14:50:13.352662 (+ 1277us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.352674 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.352694 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:13.353366 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.353372 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.353579 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.353746 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:13.353932 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.355146 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.356739 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.357315 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.357433 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.357670 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.357889 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.358008 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1157,"thread_start_us":131,"threads_started":1}
I20260928 14:50:13.358198 894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.352401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:46001 (local address 127.0.0.1:39206)
0928 14:50:13.352849 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.352863 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.352945 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.353577 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.353579 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.353598 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:13.353661 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.353665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.354745 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.354751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.356527 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.356538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.357468 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.357471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.357653 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.357711 (+ 58us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.357848 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.358003 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":125,"thread_start_us":110,"threads_started":1}
I20260928 14:50:13.358223 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.351225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35843 (local address 127.0.0.1:46568)
0928 14:50:13.351673 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.351737 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.351889 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.353729 (+ 1840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.353733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.353751 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:13.353844 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.353850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.355268 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.355275 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.356620 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.356636 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.357454 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.357459 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.357687 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.357716 (+ 29us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.357853 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.358007 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":101,"thread_start_us":110,"threads_started":1}
I20260928 14:50:13.358803 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.352582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39206 (local address 127.13.223.65:46001)
0928 14:50:13.352955 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.352962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.353017 (+ 55us) server_negotiation.cc:407] Connection header received
0928 14:50:13.353115 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.353120 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.353310 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.353453 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:13.353805 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.354637 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.356676 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.357317 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.357466 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.357699 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.358505 (+ 806us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.358643 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260928 14:50:13.359695 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.351434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40387 (local address 127.0.0.1:55880)
0928 14:50:13.351970 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.351984 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.352074 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.353128 (+ 1054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.353135 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.353157 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:13.353314 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.353322 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.354912 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.354915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.356527 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.356538 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.359246 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.359250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.359388 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.359422 (+ 34us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.359481 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.359552 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":195,"thread_start_us":84,"threads_started":1}
I20260928 14:50:13.360592 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.351943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55880 (local address 127.13.223.67:40387)
0928 14:50:13.352540 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.352554 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.352584 (+ 30us) server_negotiation.cc:407] Connection header received
0928 14:50:13.352672 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.352678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.352911 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.353140 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:13.353425 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.354809 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.357774 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.359096 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.360007 (+ 911us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.360215 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.360269 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.360393 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":195,"threads_started":1}
I20260928 14:50:13.362262 679 tablet_service.cc:1467] Processing CreateTablet for tablet 8048e50490164098832ad19054b8134a (DEFAULT_TABLE table=test-table [id=5ca57cc763ce40cea071a6be68b572fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:13.363538 679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048e50490164098832ad19054b8134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:13.363490 812 tablet_service.cc:1467] Processing CreateTablet for tablet 8048e50490164098832ad19054b8134a (DEFAULT_TABLE table=test-table [id=5ca57cc763ce40cea071a6be68b572fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:13.364631 812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048e50490164098832ad19054b8134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:13.365227 546 tablet_service.cc:1467] Processing CreateTablet for tablet 8048e50490164098832ad19054b8134a (DEFAULT_TABLE table=test-table [id=5ca57cc763ce40cea071a6be68b572fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:13.366410 546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8048e50490164098832ad19054b8134a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:13.370764 896 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Bootstrap starting.
I20260928 14:50:13.372582 897 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Bootstrap starting.
I20260928 14:50:13.373200 896 tablet_bootstrap.cc:654] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:13.373962 896 log.cc:824] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:13.375101 897 tablet_bootstrap.cc:654] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:13.376050 899 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Bootstrap starting.
I20260928 14:50:13.376068 897 log.cc:824] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:13.376596 896 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: No bootstrap required, opened a new log
I20260928 14:50:13.376832 896 ts_tablet_manager.cc:1397] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:50:13.378228 899 tablet_bootstrap.cc:654] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:13.379199 899 log.cc:824] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:13.380802 897 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: No bootstrap required, opened a new log
I20260928 14:50:13.381031 897 ts_tablet_manager.cc:1397] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260928 14:50:13.380916 896 raft_consensus.cc:348] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.381211 896 raft_consensus.cc:374] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:13.381263 896 raft_consensus.cc:729] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48827e1f7a6645859a8dfdc0245bf5aa, State: Initialized, Role: FOLLOWER
I20260928 14:50:13.381564 899 tablet_bootstrap.cc:492] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: No bootstrap required, opened a new log
I20260928 14:50:13.381778 899 ts_tablet_manager.cc:1397] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:50:13.381776 896 consensus_queue.cc:260] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.382891 896 ts_tablet_manager.cc:1428] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:50:13.383282 879 heartbeater.cc:503] Master 127.13.223.126:37663 was elected leader, sending a full tablet report...
I20260928 14:50:13.386149 897 raft_consensus.cc:348] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.386494 897 raft_consensus.cc:374] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:13.386619 897 raft_consensus.cc:729] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 107e019d74bb45bd88cb90f0bce30c66, State: Initialized, Role: FOLLOWER
I20260928 14:50:13.386674 899 raft_consensus.cc:348] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.386965 899 raft_consensus.cc:374] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:13.387039 899 raft_consensus.cc:729] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fb0a11b4014de7a59868ff901b05b7, State: Initialized, Role: FOLLOWER
I20260928 14:50:13.387179 897 consensus_queue.cc:260] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.387481 899 consensus_queue.cc:260] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.388257 746 heartbeater.cc:503] Master 127.13.223.126:37663 was elected leader, sending a full tablet report...
I20260928 14:50:13.388408 897 ts_tablet_manager.cc:1428] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260928 14:50:13.388408 899 ts_tablet_manager.cc:1428] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 14:50:13.388780 613 heartbeater.cc:503] Master 127.13.223.126:37663 was elected leader, sending a full tablet report...
I20260928 14:50:13.550498 905 raft_consensus.cc:482] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:13.550786 905 raft_consensus.cc:504] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.552045 905 leader_election.cc:288] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 107e019d74bb45bd88cb90f0bce30c66 (127.13.223.66:35843), 48827e1f7a6645859a8dfdc0245bf5aa (127.13.223.67:40387)
I20260928 14:50:13.556164 891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.552465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:43227 (local address 127.13.223.66:35843)
0928 14:50:13.552607 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.552613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.552799 (+ 186us) server_negotiation.cc:407] Connection header received
0928 14:50:13.552962 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.552966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.553026 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.553127 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:13.553405 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.553947 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555165 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.555573 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555703 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.555756 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.555927 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.556009 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:50:13.556315 907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.552329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:40387 (local address 127.13.223.65:48557)
0928 14:50:13.552970 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.552982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.553113 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.553372 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.553374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.553391 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:13.553522 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.553529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.554248 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.554252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.555160 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.555169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.555841 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.555938 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.555974 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.556059 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.556133 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"mutex_wait_us":306,"thread_start_us":127,"threads_started":1}
I20260928 14:50:13.556304 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.552471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:48557 (local address 127.13.223.67:40387)
0928 14:50:13.552617 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.552622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.553070 (+ 448us) server_negotiation.cc:407] Connection header received
0928 14:50:13.553220 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.553223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.553276 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.553358 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:13.553629 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.554136 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555284 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.555707 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555841 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.555889 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.556090 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.556167 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:50:13.556190 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.552329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:35843 (local address 127.13.223.65:43227)
0928 14:50:13.552704 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.552717 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:13.552852 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:13.553138 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:13.553141 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:13.553159 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:13.553296 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.553303 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.554064 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.554067 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.555040 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:13.555050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.555698 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.555702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:13.555798 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.555820 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:50:13.555912 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 14:50:13.556028 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":23,"thread_start_us":149,"threads_started":1}
W20260928 14:50:13.556988 880 tablet.cc:2369] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:13.557152 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048e50490164098832ad19054b8134a" candidate_uuid: "e6fb0a11b4014de7a59868ff901b05b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" is_pre_election: true
I20260928 14:50:13.557334 699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048e50490164098832ad19054b8134a" candidate_uuid: "e6fb0a11b4014de7a59868ff901b05b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107e019d74bb45bd88cb90f0bce30c66" is_pre_election: true
I20260928 14:50:13.557559 832 raft_consensus.cc:2454] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6fb0a11b4014de7a59868ff901b05b7 in term 0.
I20260928 14:50:13.557613 699 raft_consensus.cc:2454] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e6fb0a11b4014de7a59868ff901b05b7 in term 0.
I20260928 14:50:13.558071 499 leader_election.cc:302] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107e019d74bb45bd88cb90f0bce30c66, e6fb0a11b4014de7a59868ff901b05b7; no voters:
I20260928 14:50:13.558348 905 raft_consensus.cc:2781] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:13.558446 905 raft_consensus.cc:482] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:13.558490 905 raft_consensus.cc:3037] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:13.559621 905 raft_consensus.cc:504] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.560014 905 leader_election.cc:288] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [CANDIDATE]: Term 1 election: Requested vote from peers 107e019d74bb45bd88cb90f0bce30c66 (127.13.223.66:35843), 48827e1f7a6645859a8dfdc0245bf5aa (127.13.223.67:40387)
I20260928 14:50:13.560400 699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048e50490164098832ad19054b8134a" candidate_uuid: "e6fb0a11b4014de7a59868ff901b05b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "107e019d74bb45bd88cb90f0bce30c66"
I20260928 14:50:13.560472 832 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8048e50490164098832ad19054b8134a" candidate_uuid: "e6fb0a11b4014de7a59868ff901b05b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48827e1f7a6645859a8dfdc0245bf5aa"
I20260928 14:50:13.560552 699 raft_consensus.cc:3037] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:13.560628 832 raft_consensus.cc:3037] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:13.561785 699 raft_consensus.cc:2454] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6fb0a11b4014de7a59868ff901b05b7 in term 1.
I20260928 14:50:13.561946 832 raft_consensus.cc:2454] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e6fb0a11b4014de7a59868ff901b05b7 in term 1.
I20260928 14:50:13.562173 499 leader_election.cc:302] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 107e019d74bb45bd88cb90f0bce30c66, e6fb0a11b4014de7a59868ff901b05b7; no voters:
I20260928 14:50:13.562362 905 raft_consensus.cc:2781] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:13.562608 905 raft_consensus.cc:686] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [term 1 LEADER]: Becoming Leader. State: Replica: e6fb0a11b4014de7a59868ff901b05b7, State: Running, Role: LEADER
I20260928 14:50:13.563014 905 consensus_queue.cc:237] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } }
I20260928 14:50:13.566679 421 catalog_manager.cc:5495] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e6fb0a11b4014de7a59868ff901b05b7 (127.13.223.65). New cstate: current_term: 1 leader_uuid: "e6fb0a11b4014de7a59868ff901b05b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e6fb0a11b4014de7a59868ff901b05b7" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 46001 } health_report { overall_health: HEALTHY } } }
W20260928 14:50:13.575776 747 tablet.cc:2369] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 14:50:13.599628 614 tablet.cc:2369] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:13.606673 895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.602886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39216 (local address 127.13.223.65:46001)
0928 14:50:13.603018 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:13.603022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:13.603038 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:50:13.603140 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:13.603142 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:13.603199 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:13.603300 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:13.603560 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.604077 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.604975 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:13.605230 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:13.605314 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:13.605667 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:13.606107 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:13.606223 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:13.606443 (+ 220us) server_negotiation.cc:299] Negotiation successful
0928 14:50:13.606480 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:50:13.613511 699 raft_consensus.cc:1261] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 1 FOLLOWER]: Refusing update from remote peer e6fb0a11b4014de7a59868ff901b05b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:13.613543 832 raft_consensus.cc:1261] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 1 FOLLOWER]: Refusing update from remote peer e6fb0a11b4014de7a59868ff901b05b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:13.614365 908 consensus_queue.cc:1035] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48827e1f7a6645859a8dfdc0245bf5aa" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 40387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:13.614663 905 consensus_queue.cc:1035] T 8048e50490164098832ad19054b8134a P e6fb0a11b4014de7a59868ff901b05b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "107e019d74bb45bd88cb90f0bce30c66" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 35843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:13.622641 913 mvcc.cc:205] Tried to move back new op lower bound from 7334326327751442432 to 7334326327557251072. Current Snapshot: MvccSnapshot[applied={T|T < 7334326327751442432}]
I20260928 14:50:13.622994 912 mvcc.cc:205] Tried to move back new op lower bound from 7334326327751442432 to 7334326327557251072. Current Snapshot: MvccSnapshot[applied={T|T < 7334326327751442432}]
I20260928 14:50:13.634644 421 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:13.634907 421 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43854:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:13.637661 421 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4445496264ed42ac8febebce2ae45882: Sending DeleteTablet for 3 replicas of tablet 8048e50490164098832ad19054b8134a
I20260928 14:50:13.638571 812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048e50490164098832ad19054b8134a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:13 UTC) from {username='slave'} at 127.0.0.1:55880
I20260928 14:50:13.638578 679 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048e50490164098832ad19054b8134a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:13 UTC) from {username='slave'} at 127.0.0.1:46568
I20260928 14:50:13.639168 922 tablet_replica.cc:330] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: stopping tablet replica
I20260928 14:50:13.639539 922 raft_consensus.cc:2229] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:13.639559 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 483
I20260928 14:50:13.639645 546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8048e50490164098832ad19054b8134a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:13 UTC) from {username='slave'} at 127.0.0.1:39206
I20260928 14:50:13.640748 922 raft_consensus.cc:2258] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:13.641141 921 tablet_replica.cc:330] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: stopping tablet replica
I20260928 14:50:13.641376 921 raft_consensus.cc:2229] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:13.641623 921 raft_consensus.cc:2258] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:13.643184 922 ts_tablet_manager.cc:1905] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:13.643335 921 ts_tablet_manager.cc:1905] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:13.645972 921 ts_tablet_manager.cc:1918] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:13.645972 922 ts_tablet_manager.cc:1918] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:13.646083 922 log.cc:1196] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-1/wal/wals/8048e50490164098832ad19054b8134a
I20260928 14:50:13.646083 921 log.cc:1196] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/8048e50490164098832ad19054b8134a
I20260928 14:50:13.646446 921 ts_tablet_manager.cc:1939] T 8048e50490164098832ad19054b8134a P 48827e1f7a6645859a8dfdc0245bf5aa: Deleting consensus metadata
I20260928 14:50:13.646446 922 ts_tablet_manager.cc:1939] T 8048e50490164098832ad19054b8134a P 107e019d74bb45bd88cb90f0bce30c66: Deleting consensus metadata
W20260928 14:50:13.647455 408 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:46001 (error 108)
I20260928 14:50:13.647838 407 catalog_manager.cc:4841] TS 48827e1f7a6645859a8dfdc0245bf5aa (127.13.223.67:40387): tablet 8048e50490164098832ad19054b8134a (table test-table [id=5ca57cc763ce40cea071a6be68b572fa]) successfully deleted
I20260928 14:50:13.648069 406 catalog_manager.cc:4841] TS 107e019d74bb45bd88cb90f0bce30c66 (127.13.223.66:35843): tablet 8048e50490164098832ad19054b8134a (table test-table [id=5ca57cc763ce40cea071a6be68b572fa]) successfully deleted
I20260928 14:50:13.648418 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 616
I20260928 14:50:13.648646 892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:13.648285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:46001 (local address 127.0.0.1:39232)
0928 14:50:13.648435 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:13.648538 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:46001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 14:50:13.649343 408 catalog_manager.cc:4584] TS e6fb0a11b4014de7a59868ff901b05b7 (127.13.223.65:46001): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8048e50490164098832ad19054b8134a: Network error: Client connection negotiation failed: client connection to 127.13.223.65:46001: connect: Connection refused (error 111)
I20260928 14:50:13.655082 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 749
I20260928 14:50:13.661149 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 389
2026-09-28T14:50:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1297 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:13 dist-test-slave-gkhx krb5kdc[926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:13 dist-test-slave-gkhx krb5kdc[926](info): set up 2 sockets
Sep 28 14:50:13 dist-test-slave-gkhx krb5kdc[926](info): commencing operation
krb5kdc: starting...
W20260928 14:50:14.772782 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:14 dist-test-slave-gkhx krb5kdc[926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:14.911697 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41805
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:41805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:15.028692 939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:15.029013 939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:15.029058 939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:15.033197 939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:15.033277 939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:15.033298 939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:15.033316 939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:15.033332 939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:15.039156 939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:41805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:41805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:15.040304 939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:15.041338 939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:15.050731 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.050818 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:15.051007 939 server_base.cc:1034] running on GCE node
W20260928 14:50:15.050731 944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:15.051862 939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:15.053223 939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:15.054414 939 hybrid_clock.cc:648] HybridClock initialized: now 1790607015054372 us; error 59 us; skew 500 ppm
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:15.057791 939 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:15.058954 939 webserver.cc:460] Webserver started at http://127.13.223.126:44213/ using document root <none> and password file <none>
I20260928 14:50:15.059577 939 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:15.059713 939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:15.059965 939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:15.062255 939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "7879352a2c2d4892bd0d5c32c58c2119"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "f4ddc3418f4e1e454d4ba3300852b04c"
server_key_iv: "4f29c6d497c9edb0ee78fa4bafd145dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.062911 939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "7879352a2c2d4892bd0d5c32c58c2119"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "f4ddc3418f4e1e454d4ba3300852b04c"
server_key_iv: "4f29c6d497c9edb0ee78fa4bafd145dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.067133 939 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:15.070084 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.071559 939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:15.071676 939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "7879352a2c2d4892bd0d5c32c58c2119"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "f4ddc3418f4e1e454d4ba3300852b04c"
server_key_iv: "4f29c6d497c9edb0ee78fa4bafd145dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.071784 939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:15.083192 939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:15.086688 939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:15.086853 939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:15.094732 939 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:41805
I20260928 14:50:15.094762 1006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:41805 every 8 connection(s)
I20260928 14:50:15.095758 939 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:15.098887 1007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:15.099339 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 939
I20260928 14:50:15.099525 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:15.099810 14205 external_mini_cluster.cc:1428] Setting key def7e96ba564346f6761891a22789a66
I20260928 14:50:15.105834 1007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: Bootstrap starting.
I20260928 14:50:15.108248 1007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: Neither blocks nor log segments found. Creating new log.
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:15.109229 1007 log.cc:824] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:15.111415 1007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: No bootstrap required, opened a new log
I20260928 14:50:15.114781 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.101305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57412 (local address 127.13.223.126:41805)
0928 14:50:15.101803 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.101819 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.101856 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:50:15.102538 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.102623 (+ 85us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.103063 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.103431 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:15.104302 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.105322 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.106256 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.106821 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.107008 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.109787 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:15.109792 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:15.109807 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:15.109835 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:15.111781 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.112253 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.112259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.112264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.112377 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.112679 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.112683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.112685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.113112 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:15.113331 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.113630 (+ 299us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.113932 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":128,"threads_started":1}
I20260928 14:50:15.115238 1007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } }
I20260928 14:50:15.115501 1007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:15.115576 1007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7879352a2c2d4892bd0d5c32c58c2119, State: Initialized, Role: FOLLOWER
I20260928 14:50:15.116055 1007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } }
I20260928 14:50:15.116211 1007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:15.116295 1007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:15.116396 1007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:15.117507 1007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } }
I20260928 14:50:15.117909 1007 leader_election.cc:302] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7879352a2c2d4892bd0d5c32c58c2119; no voters:
I20260928 14:50:15.118309 1007 leader_election.cc:288] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:15.118498 1012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:15.118868 1012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [term 1 LEADER]: Becoming Leader. State: Replica: 7879352a2c2d4892bd0d5c32c58c2119, State: Running, Role: LEADER
I20260928 14:50:15.119515 1012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } }
I20260928 14:50:15.119966 1007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:15.120692 1012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7879352a2c2d4892bd0d5c32c58c2119. Latest consensus state: current_term: 1 leader_uuid: "7879352a2c2d4892bd0d5c32c58c2119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } } }
I20260928 14:50:15.120816 1012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:15.121204 1013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7879352a2c2d4892bd0d5c32c58c2119" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7879352a2c2d4892bd0d5c32c58c2119" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 41805 } } }
I20260928 14:50:15.121304 1013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119 [sys.catalog]: This master's current role is: LEADER
W20260928 14:50:15.125420 1027 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:15.125540 1027 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:15.125626 1020 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:15.126772 1020 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:15.134313 1020 catalog_manager.cc:1347] Generated new cluster ID: 3dbc88155e7a443abdc2db5ac00c41b0
I20260928 14:50:15.134418 1020 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:15.145756 1020 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:15.147027 1020 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:15.159819 1020 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: Generated new TSK 0
I20260928 14:50:15.160713 1020 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.13.223.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
Entry for principal kudu/127.13.223.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab.
I20260928 14:50:15.200096 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:15.325210 1033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:15.325524 1033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:15.325572 1033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:15.329893 1033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:15.329975 1033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:15.330039 1033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:15.335707 1033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.65.keytab
--principal=kudu/127.13.223.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:15.336853 1033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:15.338063 1033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:15.348829 1039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.348811 1038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.348811 1041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:15.349329 1033 server_base.cc:1034] running on GCE node
I20260928 14:50:15.349776 1033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:15.350495 1033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:15.351739 1033 hybrid_clock.cc:648] HybridClock initialized: now 1790607015351691 us; error 63 us; skew 500 ppm
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:15.354777 1033 init.cc:377] Logged in from keytab as kudu/127.13.223.65@KRBTEST.COM (short username kudu)
I20260928 14:50:15.355743 1033 webserver.cc:460] Webserver started at http://127.13.223.65:46099/ using document root <none> and password file <none>
I20260928 14:50:15.356324 1033 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:15.356387 1033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:15.356564 1033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:15.358845 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "22bb859b7e7d4a44b0d25b10642f282f"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "aaff5dffbb7adb5b5024c9a73c36d349"
server_key_iv: "285a639650a4be369446a565cfb3c1e7"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.359460 1033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "22bb859b7e7d4a44b0d25b10642f282f"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "aaff5dffbb7adb5b5024c9a73c36d349"
server_key_iv: "285a639650a4be369446a565cfb3c1e7"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.363803 1033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:15.366681 1048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.368508 1033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:15.368686 1033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "22bb859b7e7d4a44b0d25b10642f282f"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "aaff5dffbb7adb5b5024c9a73c36d349"
server_key_iv: "285a639650a4be369446a565cfb3c1e7"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.368835 1033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:15.407123 1033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:15.411203 1033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:15.411460 1033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:15.412170 1033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:15.413163 1033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:15.413218 1033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.413288 1033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:15.413326 1033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.423944 1033 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:38143
I20260928 14:50:15.423988 1163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:38143 every 8 connection(s)
I20260928 14:50:15.424963 1033 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:15.429339 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1033
I20260928 14:50:15.429476 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:15.429848 14205 external_mini_cluster.cc:1428] Setting key 80d577d59150f1717a0ee38d161cf963
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.65@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.13.223.66@KRBTEST.COM; defaulting to no policy
I20260928 14:50:15.441928 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.427126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:37699 (local address 127.13.223.126:41805)
0928 14:50:15.427291 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.427295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.428158 (+ 863us) server_negotiation.cc:407] Connection header received
0928 14:50:15.429894 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.429898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.429967 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.430095 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:15.431711 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.432316 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.434270 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.434645 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.434774 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.437780 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:15.437784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:15.437787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:15.437823 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:15.439477 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.440077 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.440080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.440082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.440166 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.440554 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.440558 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.440560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.440828 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:15.440937 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.441538 (+ 601us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.441740 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:50:15.442932 1166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.427489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41805 (local address 127.13.223.65:37699)
0928 14:50:15.427986 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.428040 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.429655 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.430303 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.430311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.430868 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:15.431547 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.431564 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.432475 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.432479 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.434098 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.434114 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.434918 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.434921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.435140 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.435838 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:15.435856 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:15.437580 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:15.439625 (+ 2045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.439632 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.439652 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.439953 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.440261 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.440264 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.440266 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.440445 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.440938 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:15.440949 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:15.441265 (+ 316us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.441469 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.441880 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":131,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.66@KRBTEST.COM" created.
I20260928 14:50:15.444629 1164 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41805
I20260928 14:50:15.444918 1164 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:15.445536 1164 heartbeater.cc:511] Master 127.13.223.126:41805 requested a full tablet report, sending...
I20260928 14:50:15.447686 971 ts_manager.cc:194] Registered new tserver with Master: 22bb859b7e7d4a44b0d25b10642f282f (127.13.223.65:38143)
I20260928 14:50:15.449074 971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.65@KRBTEST.COM'} at 127.13.223.65:37699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
Entry for principal kudu/127.13.223.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab.
I20260928 14:50:15.466392 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:15.584371 1169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:15.584731 1169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:15.584808 1169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:15.589102 1169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:15.589227 1169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:15.589339 1169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:15.595429 1169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.66.keytab
--principal=kudu/127.13.223.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:15.596710 1169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:15.597836 1169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:15.608951 1174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.608930 1177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.608929 1175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:15.609404 1169 server_base.cc:1034] running on GCE node
I20260928 14:50:15.609939 1169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:15.610824 1169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:15.612025 1169 hybrid_clock.cc:648] HybridClock initialized: now 1790607015611979 us; error 63 us; skew 500 ppm
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:15.615383 1169 init.cc:377] Logged in from keytab as kudu/127.13.223.66@KRBTEST.COM (short username kudu)
I20260928 14:50:15.616500 1169 webserver.cc:460] Webserver started at http://127.13.223.66:42587/ using document root <none> and password file <none>
I20260928 14:50:15.617172 1169 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:15.617245 1169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:15.617486 1169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:15.620149 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "1f6523760f5046319b1780e290917953"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9bb229611bcf903bac884975ab82e03d"
server_key_iv: "f8aab76b414c7368d6c1c2f97b26a386"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.620965 1169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "1f6523760f5046319b1780e290917953"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9bb229611bcf903bac884975ab82e03d"
server_key_iv: "f8aab76b414c7368d6c1c2f97b26a386"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.625631 1169 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:15.628779 1184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.630440 1169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:15.630679 1169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "1f6523760f5046319b1780e290917953"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9bb229611bcf903bac884975ab82e03d"
server_key_iv: "f8aab76b414c7368d6c1c2f97b26a386"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.630822 1169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:15.648058 1169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:15.651975 1169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:15.652196 1169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:15.652925 1169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:15.653955 1169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:15.654026 1169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.654105 1169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:15.654148 1169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.664907 1169 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:46771
I20260928 14:50:15.664927 1300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:46771 every 8 connection(s)
I20260928 14:50:15.665982 1169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:15.674474 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1169
I20260928 14:50:15.674661 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:15.675014 14205 external_mini_cluster.cc:1428] Setting key b198034b31e5ba1186a2635f81a8ca17
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.66@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:15.682787 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.668258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:44719 (local address 127.13.223.126:41805)
0928 14:50:15.668434 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.668438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.669318 (+ 880us) server_negotiation.cc:407] Connection header received
0928 14:50:15.671020 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.671024 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.671097 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.671215 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:15.672640 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.673365 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.674954 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.675280 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.675422 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.678392 (+ 2970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:15.678396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:15.678399 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:15.678435 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:15.680145 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.680880 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.680885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.680887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.680989 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.681391 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.681397 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.681399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.681635 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:15.681757 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.682386 (+ 629us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.682593 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260928 14:50:15.683866 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.668555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41805 (local address 127.13.223.66:44719)
0928 14:50:15.669113 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.669167 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.670750 (+ 1583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.671423 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.671431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.671856 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:15.672462 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.672480 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.673516 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.673520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.674795 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.674807 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.675463 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.675470 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.675728 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.676447 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:15.676476 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:15.678188 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:15.680298 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.680306 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.680326 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.680745 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.681093 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.681097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.681099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.681282 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.681757 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:15.681767 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:15.682107 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.682336 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.682772 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":141,"threads_started":1}
WARNING: no policy specified for kudu/127.13.223.67@KRBTEST.COM; defaulting to no policy
I20260928 14:50:15.685331 1301 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41805
I20260928 14:50:15.685608 1301 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:15.686233 1301 heartbeater.cc:511] Master 127.13.223.126:41805 requested a full tablet report, sending...
I20260928 14:50:15.687526 971 ts_manager.cc:194] Registered new tserver with Master: 1f6523760f5046319b1780e290917953 (127.13.223.66:46771)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.67@KRBTEST.COM" created.
I20260928 14:50:15.688446 971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.66@KRBTEST.COM'} at 127.13.223.66:44719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
Entry for principal kudu/127.13.223.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab.
I20260928 14:50:15.707351 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:38949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:15.828917 1306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:15.829221 1306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:15.829267 1306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:15.833384 1306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:15.833453 1306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:15.833516 1306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:15.839200 1306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:38949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.67.keytab
--principal=kudu/127.13.223.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:41805
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:15.840361 1306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:15.841442 1306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:15.851883 1314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.852041 1311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:15.851883 1312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:15.852156 1306 server_base.cc:1034] running on GCE node
I20260928 14:50:15.852591 1306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:15.853314 1306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:15.854518 1306 hybrid_clock.cc:648] HybridClock initialized: now 1790607015854497 us; error 36 us; skew 500 ppm
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:15.857702 1306 init.cc:377] Logged in from keytab as kudu/127.13.223.67@KRBTEST.COM (short username kudu)
I20260928 14:50:15.858762 1306 webserver.cc:460] Webserver started at http://127.13.223.67:36015/ using document root <none> and password file <none>
I20260928 14:50:15.859429 1306 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:15.859529 1306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:15.859774 1306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:15.862095 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "e909608dd84b4d88b67cad50e57c8671"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9640cc8625c6bf6c14a6bf815c9ef99c"
server_key_iv: "b814cb673c5a8267bda66edcacd937d8"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.862776 1306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "e909608dd84b4d88b67cad50e57c8671"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9640cc8625c6bf6c14a6bf815c9ef99c"
server_key_iv: "b814cb673c5a8267bda66edcacd937d8"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.867082 1306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:15.869933 1322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.871529 1306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:15.871686 1306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "e909608dd84b4d88b67cad50e57c8671"
format_stamp: "Formatted at 2026-09-28 14:50:15 on dist-test-slave-gkhx"
server_key: "9640cc8625c6bf6c14a6bf815c9ef99c"
server_key_iv: "b814cb673c5a8267bda66edcacd937d8"
server_key_version: "encryptionkey@0"
I20260928 14:50:15.871824 1306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:15.900458 1306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:15.904057 1306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:15.904269 1306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:15.904969 1306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:15.905828 1306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:15.905875 1306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.905951 1306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:15.905987 1306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:15.915166 1306 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:46271
I20260928 14:50:15.915249 1436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:46271 every 8 connection(s)
I20260928 14:50:15.916154 1306 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:15.925513 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1306
I20260928 14:50:15.925678 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:15.926124 14205 external_mini_cluster.cc:1428] Setting key bc6ae6ac0fec95463e8c95ab76b4d3b6
Sep 28 14:50:15 dist-test-slave-gkhx krb5kdc[926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607015, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.67@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:15.933123 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.918258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:32949 (local address 127.13.223.126:41805)
0928 14:50:15.918541 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.918585 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.919592 (+ 1007us) server_negotiation.cc:407] Connection header received
0928 14:50:15.921224 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.921228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.921288 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.921394 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:15.922812 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.923419 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.924802 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.925143 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.925277 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.928926 (+ 3649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:15.928933 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:15.928940 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:15.928983 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:15.930863 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.931408 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.931411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.931412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.931493 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.931811 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.931813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.931815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.932013 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:15.932179 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.932783 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.932918 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174}
I20260928 14:50:15.934079 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.918856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:41805 (local address 127.13.223.67:32949)
0928 14:50:15.919400 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.919455 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.920976 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.921608 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.921625 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.922106 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:15.922662 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.922677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.923563 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.923566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.924661 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.924673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.925382 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.925387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.925672 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.926473 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:15.926500 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:15.928695 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:15.930987 (+ 2292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.930993 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.931010 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.931306 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.931575 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:15.931577 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:15.931579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:15.931724 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:15.932235 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:15.932241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:15.932541 (+ 300us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.932715 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.933097 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":129,"threads_started":1}
I20260928 14:50:15.935598 1437 heartbeater.cc:347] Connected to a master server at 127.13.223.126:41805
I20260928 14:50:15.935904 1437 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:15.936452 1437 heartbeater.cc:511] Master 127.13.223.126:41805 requested a full tablet report, sending...
I20260928 14:50:15.937678 971 ts_manager.cc:194] Registered new tserver with Master: e909608dd84b4d88b67cad50e57c8671 (127.13.223.67:46271)
I20260928 14:50:15.938347 971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.13.223.67@KRBTEST.COM'} at 127.13.223.67:32949
I20260928 14:50:15.942601 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:15.962698 1010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.944911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57416 (local address 127.13.223.126:41805)
0928 14:50:15.945180 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.945185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.945312 (+ 127us) server_negotiation.cc:407] Connection header received
0928 14:50:15.945489 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.945492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.945575 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.945662 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:15.946394 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.947004 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.947872 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.948113 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.948224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.958964 (+ 10740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:15.958969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:15.958971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:15.959010 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:15.960678 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.961208 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.961212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.961214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.961313 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:15.961617 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:15.961620 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:15.961622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:15.961874 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:15.962035 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.962344 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.962471 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177}
I20260928 14:50:15.966401 971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:15.969637 971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:15.986254 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.979717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54708 (local address 127.13.223.67:46271)
0928 14:50:15.981180 (+ 1463us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.981192 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.981211 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:50:15.981277 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.981280 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.981448 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.981764 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:15.981931 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.982803 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.984486 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.985428 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985555 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.985739 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.985938 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.986073 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1254,"thread_start_us":117,"threads_started":1}
I20260928 14:50:15.986238 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.980360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:46771 (local address 127.0.0.1:33238)
0928 14:50:15.980830 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.980844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.980926 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.981765 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.981767 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.981782 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:15.981861 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.981865 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.983368 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.983372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.984595 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.984605 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.985809 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.985931 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.985957 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.986029 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.986100 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"mutex_wait_us":30,"thread_start_us":98,"threads_started":1}
I20260928 14:50:15.986478 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.979637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:38143 (local address 127.0.0.1:35484)
0928 14:50:15.980589 (+ 952us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.980602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.980702 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.981472 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.981478 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.981505 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:15.981850 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.981855 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.983368 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.983372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.984895 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.984902 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.986044 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.986047 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.986140 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.986166 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.986232 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.986331 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"mutex_wait_us":428,"thread_start_us":121,"threads_started":1}
I20260928 14:50:15.986250 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.979581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:46271 (local address 127.0.0.1:54708)
0928 14:50:15.980033 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:15.980086 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:15.980212 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:15.981603 (+ 1391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:15.981617 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:15.981630 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:15.981758 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.981766 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.982926 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.982931 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.984361 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:15.984371 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.985576 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:15.985710 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.985764 (+ 54us) client_negotiation.cc:769] Sending connection context
0928 14:50:15.985886 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 14:50:15.986032 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":17,"thread_start_us":98,"threads_started":1}
I20260928 14:50:15.987002 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.980534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33238 (local address 127.13.223.66:46771)
0928 14:50:15.981194 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.981205 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.981225 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:15.981302 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.981307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.981489 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.981622 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:15.982019 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.983221 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.984730 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.985543 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985636 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.985747 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.986726 (+ 979us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.986844 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":123,"threads_started":1}
I20260928 14:50:15.987062 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:15.980455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35484 (local address 127.13.223.65:38143)
0928 14:50:15.980941 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:15.980949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:15.980973 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:50:15.981056 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:15.981062 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:15.981300 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:15.981486 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:15.981965 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.983221 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985013 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:15.985715 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:15.985814 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:15.985985 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:15.986760 (+ 775us) server_negotiation.cc:299] Negotiation successful
0928 14:50:15.986909 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":95,"threads_started":1}
I20260928 14:50:15.990047 1234 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 (DEFAULT_TABLE table=test-table [id=b1f86a1294b448f686ca530d569198b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:15.990069 1097 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 (DEFAULT_TABLE table=test-table [id=b1f86a1294b448f686ca530d569198b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:15.990072 1371 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 (DEFAULT_TABLE table=test-table [id=b1f86a1294b448f686ca530d569198b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:15.991171 1234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4393e95e34a0a9328fd886dcbadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:15.991171 1371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4393e95e34a0a9328fd886dcbadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:15.991171 1097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4393e95e34a0a9328fd886dcbadb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:15.998113 1455 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Bootstrap starting.
I20260928 14:50:15.999348 1457 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Bootstrap starting.
I20260928 14:50:16.000541 1455 tablet_bootstrap.cc:654] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:16.001309 1457 tablet_bootstrap.cc:654] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:16.001762 1455 log.cc:824] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:16.002509 1457 log.cc:824] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:16.002745 1456 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: Bootstrap starting.
I20260928 14:50:16.004284 1455 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: No bootstrap required, opened a new log
I20260928 14:50:16.004598 1455 ts_tablet_manager.cc:1397] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:50:16.004921 1457 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: No bootstrap required, opened a new log
I20260928 14:50:16.005131 1457 ts_tablet_manager.cc:1397] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 14:50:16.005182 1456 tablet_bootstrap.cc:654] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:16.006073 1456 log.cc:824] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:16.008121 1455 raft_consensus.cc:348] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.008381 1455 raft_consensus.cc:374] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:16.008436 1455 raft_consensus.cc:729] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f6523760f5046319b1780e290917953, State: Initialized, Role: FOLLOWER
I20260928 14:50:16.008446 1456 tablet_bootstrap.cc:492] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: No bootstrap required, opened a new log
I20260928 14:50:16.008658 1456 ts_tablet_manager.cc:1397] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 14:50:16.008913 1455 consensus_queue.cc:260] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.009204 1457 raft_consensus.cc:348] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.009764 1457 raft_consensus.cc:374] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:16.009814 1457 raft_consensus.cc:729] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e909608dd84b4d88b67cad50e57c8671, State: Initialized, Role: FOLLOWER
I20260928 14:50:16.010015 1455 ts_tablet_manager.cc:1428] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 14:50:16.010308 1457 consensus_queue.cc:260] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.010383 1301 heartbeater.cc:503] Master 127.13.223.126:41805 was elected leader, sending a full tablet report...
I20260928 14:50:16.011169 1437 heartbeater.cc:503] Master 127.13.223.126:41805 was elected leader, sending a full tablet report...
I20260928 14:50:16.011596 1457 ts_tablet_manager.cc:1428] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 14:50:16.013453 1456 raft_consensus.cc:348] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.013811 1456 raft_consensus.cc:374] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:16.013888 1456 raft_consensus.cc:729] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22bb859b7e7d4a44b0d25b10642f282f, State: Initialized, Role: FOLLOWER
I20260928 14:50:16.014379 1456 consensus_queue.cc:260] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.015177 1164 heartbeater.cc:503] Master 127.13.223.126:41805 was elected leader, sending a full tablet report...
I20260928 14:50:16.015604 1456 ts_tablet_manager.cc:1428] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 14:50:16.063712 1462 raft_consensus.cc:482] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:16.063995 1462 raft_consensus.cc:504] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.065668 1462 leader_election.cc:288] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22bb859b7e7d4a44b0d25b10642f282f (127.13.223.65:38143), 1f6523760f5046319b1780e290917953 (127.13.223.66:46771)
I20260928 14:50:16.071943 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.065886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:38143 (local address 127.13.223.67:44949)
0928 14:50:16.066252 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.066430 (+ 178us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:16.066590 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:16.067128 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:16.067131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:16.067412 (+ 281us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:16.067533 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.067539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.068534 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.068538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.070127 (+ 1589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.070140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.071354 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.071357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.071507 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.071543 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 14:50:16.071630 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 14:50:16.071738 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":103}
I20260928 14:50:16.072402 1452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.066241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44949 (local address 127.13.223.65:38143)
0928 14:50:16.066370 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.066376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.066507 (+ 131us) server_negotiation.cc:407] Connection header received
0928 14:50:16.066692 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.066697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.066760 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.066834 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:16.067692 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.068413 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.070319 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.071219 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.071385 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.071480 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.072177 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.072275 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 14:50:16.072928 1464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.066056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:46771 (local address 127.13.223.67:54971)
0928 14:50:16.066523 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.066543 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:16.066701 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:16.067055 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:16.067061 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:16.067095 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:16.067289 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.067298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.068392 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.068398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.070127 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.070140 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.072461 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.072465 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.072575 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.072605 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 14:50:16.072687 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 14:50:16.072770 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"mutex_wait_us":69,"thread_start_us":133,"threads_started":1}
I20260928 14:50:16.073076 1117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4393e95e34a0a9328fd886dcbadb6" candidate_uuid: "e909608dd84b4d88b67cad50e57c8671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22bb859b7e7d4a44b0d25b10642f282f" is_pre_election: true
I20260928 14:50:16.073221 1453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.066314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:54971 (local address 127.13.223.66:46771)
0928 14:50:16.066792 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.066799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.066817 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:16.066870 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.066873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.066922 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.067018 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:16.067451 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.068262 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.071554 (+ 3292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.072302 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.072458 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.072531 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.072727 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.072841 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392}
I20260928 14:50:16.073437 1117 raft_consensus.cc:2454] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e909608dd84b4d88b67cad50e57c8671 in term 0.
I20260928 14:50:16.074064 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4393e95e34a0a9328fd886dcbadb6" candidate_uuid: "e909608dd84b4d88b67cad50e57c8671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6523760f5046319b1780e290917953" is_pre_election: true
I20260928 14:50:16.074397 1254 raft_consensus.cc:2454] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e909608dd84b4d88b67cad50e57c8671 in term 0.
I20260928 14:50:16.074963 1326 leader_election.cc:302] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1f6523760f5046319b1780e290917953, 22bb859b7e7d4a44b0d25b10642f282f, e909608dd84b4d88b67cad50e57c8671; no voters:
I20260928 14:50:16.075579 1462 raft_consensus.cc:2781] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:16.075719 1462 raft_consensus.cc:482] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:16.075793 1462 raft_consensus.cc:3037] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:16.077085 1462 raft_consensus.cc:504] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.077488 1462 leader_election.cc:288] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [CANDIDATE]: Term 1 election: Requested vote from peers 22bb859b7e7d4a44b0d25b10642f282f (127.13.223.65:38143), 1f6523760f5046319b1780e290917953 (127.13.223.66:46771)
I20260928 14:50:16.077968 1254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4393e95e34a0a9328fd886dcbadb6" candidate_uuid: "e909608dd84b4d88b67cad50e57c8671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f6523760f5046319b1780e290917953"
I20260928 14:50:16.077987 1117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5d4393e95e34a0a9328fd886dcbadb6" candidate_uuid: "e909608dd84b4d88b67cad50e57c8671" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22bb859b7e7d4a44b0d25b10642f282f"
I20260928 14:50:16.078123 1117 raft_consensus.cc:3037] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:16.078123 1254 raft_consensus.cc:3037] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:16.079588 1117 raft_consensus.cc:2454] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e909608dd84b4d88b67cad50e57c8671 in term 1.
I20260928 14:50:16.079587 1254 raft_consensus.cc:2454] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e909608dd84b4d88b67cad50e57c8671 in term 1.
I20260928 14:50:16.080000 1326 leader_election.cc:302] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f6523760f5046319b1780e290917953, e909608dd84b4d88b67cad50e57c8671; no voters:
I20260928 14:50:16.080210 1462 raft_consensus.cc:2781] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:16.080461 1462 raft_consensus.cc:686] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 1 LEADER]: Becoming Leader. State: Replica: e909608dd84b4d88b67cad50e57c8671, State: Running, Role: LEADER
I20260928 14:50:16.080834 1462 consensus_queue.cc:237] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } }
I20260928 14:50:16.084445 971 catalog_manager.cc:5495] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 reported cstate change: term changed from 0 to 1, leader changed from <none> to e909608dd84b4d88b67cad50e57c8671 (127.13.223.67). New cstate: current_term: 1 leader_uuid: "e909608dd84b4d88b67cad50e57c8671" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e909608dd84b4d88b67cad50e57c8671" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 46271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:16.113340 1450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.109362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54722 (local address 127.13.223.67:46271)
0928 14:50:16.109514 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.109521 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.109540 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:50:16.109636 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.109639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.109697 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.109821 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:16.110092 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.110744 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.111669 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.111908 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.112049 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.112407 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:16.112881 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:16.113025 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.113068 (+ 43us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.113135 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 14:50:16.120637 1117 raft_consensus.cc:1261] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 1 FOLLOWER]: Refusing update from remote peer e909608dd84b4d88b67cad50e57c8671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:16.120636 1254 raft_consensus.cc:1261] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 1 FOLLOWER]: Refusing update from remote peer e909608dd84b4d88b67cad50e57c8671: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:16.121853 1465 consensus_queue.cc:1035] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f6523760f5046319b1780e290917953" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 46771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:16.122296 1462 consensus_queue.cc:1035] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22bb859b7e7d4a44b0d25b10642f282f" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 38143 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:16.132822 1469 mvcc.cc:205] Tried to move back new op lower bound from 7334326338020413440 to 7334326337870159872. Current Snapshot: MvccSnapshot[applied={T|T < 7334326338020413440}]
I20260928 14:50:16.144136 970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:16.144335 970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57416:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:16.147549 970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7879352a2c2d4892bd0d5c32c58c2119: Sending DeleteTablet for 3 replicas of tablet b5d4393e95e34a0a9328fd886dcbadb6
I20260928 14:50:16.148684 1097 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:16 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:35484
I20260928 14:50:16.148705 1371 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:16 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:54708
I20260928 14:50:16.148713 1234 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4393e95e34a0a9328fd886dcbadb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:16 UTC) from {username='kudu', principal='kudu/127.13.223.126@KRBTEST.COM'} at 127.0.0.1:33238
I20260928 14:50:16.149350 1480 tablet_replica.cc:330] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f: stopping tablet replica
I20260928 14:50:16.149696 1481 tablet_replica.cc:330] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: stopping tablet replica
I20260928 14:50:16.149766 1480 raft_consensus.cc:2229] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:16.150058 1479 tablet_replica.cc:330] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: stopping tablet replica
I20260928 14:50:16.150224 1480 raft_consensus.cc:2258] T b5d4393e95e34a0a9328fd886dcbadb6 P 22bb859b7e7d4a44b0d25b10642f282f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:16.150266 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1033
I20260928 14:50:16.150712 1481 raft_consensus.cc:2229] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:16.151053 1479 raft_consensus.cc:2229] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:16.151516 1479 raft_consensus.cc:2258] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:16.152058 1481 raft_consensus.cc:2258] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:16.154235 1481 ts_tablet_manager.cc:1905] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:16.157650 1481 ts_tablet_manager.cc:1918] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:16.157800 1481 log.cc:1196] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/b5d4393e95e34a0a9328fd886dcbadb6
I20260928 14:50:16.158177 1481 ts_tablet_manager.cc:1939] T b5d4393e95e34a0a9328fd886dcbadb6 P e909608dd84b4d88b67cad50e57c8671: Deleting consensus metadata
W20260928 14:50:16.158635 956 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:38143 (error 108)
I20260928 14:50:16.159608 1479 ts_tablet_manager.cc:1905] T b5d4393e95e34a0a9328fd886dcbadb6 P 1f6523760f5046319b1780e290917953: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:16.159674 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1169
I20260928 14:50:16.160957 958 catalog_manager.cc:4841] TS e909608dd84b4d88b67cad50e57c8671 (127.13.223.67:46271): tablet b5d4393e95e34a0a9328fd886dcbadb6 (table test-table [id=b1f86a1294b448f686ca530d569198b9]) successfully deleted
I20260928 14:50:16.162789 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.162349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:38143 (local address 127.0.0.1:35492)
0928 14:50:16.162541 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.162679 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:38143: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260928 14:50:16.163121 956 catalog_manager.cc:4584] TS 22bb859b7e7d4a44b0d25b10642f282f (127.13.223.65:38143): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b5d4393e95e34a0a9328fd886dcbadb6: Network error: Client connection negotiation failed: client connection to 127.13.223.65:38143: connect: Connection refused (error 111)
I20260928 14:50:16.167958 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1306
I20260928 14:50:16.169401 1451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.168134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:46771 (local address 127.0.0.1:33252)
0928 14:50:16.169184 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.169268 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:46771: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":928}
I20260928 14:50:16.174360 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 939
2026-09-28T14:50:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2515 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T14:50:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:16Z Disabled control of system clock
I20260928 14:50:16.208240 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44245
--webserver_interface=127.13.223.126
--webserver_port=0
--builtin_ntp_servers=127.13.223.84:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:16.337324 1487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:16.337652 1487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:16.337690 1487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:16.342103 1487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:16.342200 1487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:16.342216 1487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:16.342234 1487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:16.342250 1487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:16.348803 1487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44245
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:16.350183 1487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:16.351816 1487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:16.361766 1495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:16.361860 1493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.361931 1487 server_base.cc:1034] running on GCE node
W20260928 14:50:16.361860 1492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.362921 1487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:16.364277 1487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:16.365495 1487 hybrid_clock.cc:648] HybridClock initialized: now 1790607016365464 us; error 36 us; skew 500 ppm
I20260928 14:50:16.367969 1487 webserver.cc:460] Webserver started at http://127.13.223.126:43525/ using document root <none> and password file <none>
I20260928 14:50:16.368729 1487 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:16.368808 1487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:16.369063 1487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:16.371614 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "f398fa0c30f740e9b5e6f0b58b449ec4"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "c682e307a738dc085cfd4ed6b918376b"
server_key_iv: "81285fb84d1af25f62a0c31375dd9009"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.372321 1487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "f398fa0c30f740e9b5e6f0b58b449ec4"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "c682e307a738dc085cfd4ed6b918376b"
server_key_iv: "81285fb84d1af25f62a0c31375dd9009"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.376957 1487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:16.380278 1501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.382067 1487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:50:16.382233 1487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "f398fa0c30f740e9b5e6f0b58b449ec4"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "c682e307a738dc085cfd4ed6b918376b"
server_key_iv: "81285fb84d1af25f62a0c31375dd9009"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.382445 1487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:16.395124 1487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:16.395993 1487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:16.396251 1487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:16.404816 1487 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44245
I20260928 14:50:16.404845 1553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44245 every 8 connection(s)
I20260928 14:50:16.406023 1487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:16.409905 1554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:16.416266 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1487
I20260928 14:50:16.416419 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:16.416815 14205 external_mini_cluster.cc:1428] Setting key eca8c92d8d12f62276d764fc93321d41
I20260928 14:50:16.417263 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: Bootstrap starting.
I20260928 14:50:16.420397 1554 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:16.421399 1554 log.cc:824] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:16.423969 1554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: No bootstrap required, opened a new log
I20260928 14:50:16.428706 1554 raft_consensus.cc:348] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } }
I20260928 14:50:16.428946 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.418692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49992 (local address 127.13.223.126:44245)
0928 14:50:16.419477 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.419506 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.419555 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:50:16.420578 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.420607 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.421206 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.421634 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:16.422098 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.423559 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.424619 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.425320 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.425528 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.426462 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:16.426469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:16.426488 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:16.427091 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:16.427173 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:16.427210 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:16.427329 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.427532 (+ 203us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.427907 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":145,"threads_started":1}
I20260928 14:50:16.429033 1554 raft_consensus.cc:374] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:16.429083 1554 raft_consensus.cc:729] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f398fa0c30f740e9b5e6f0b58b449ec4, State: Initialized, Role: FOLLOWER
I20260928 14:50:16.429570 1554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } }
I20260928 14:50:16.429733 1554 raft_consensus.cc:388] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:16.429796 1554 raft_consensus.cc:482] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:16.429891 1554 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:16.431003 1554 raft_consensus.cc:504] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } }
I20260928 14:50:16.431375 1554 leader_election.cc:302] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f398fa0c30f740e9b5e6f0b58b449ec4; no voters:
I20260928 14:50:16.431712 1554 leader_election.cc:288] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:16.431829 1559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:16.432147 1559 raft_consensus.cc:686] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [term 1 LEADER]: Becoming Leader. State: Replica: f398fa0c30f740e9b5e6f0b58b449ec4, State: Running, Role: LEADER
I20260928 14:50:16.432523 1559 consensus_queue.cc:237] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } }
I20260928 14:50:16.433064 1554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:16.434173 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } } }
I20260928 14:50:16.434329 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:16.434628 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f398fa0c30f740e9b5e6f0b58b449ec4. Latest consensus state: current_term: 1 leader_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f398fa0c30f740e9b5e6f0b58b449ec4" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44245 } } }
I20260928 14:50:16.434716 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:16.434863 1568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:16.438300 1568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:16.445206 1568 catalog_manager.cc:1347] Generated new cluster ID: 60bdba6a2d3342db92b994bf03070eae
I20260928 14:50:16.445307 1568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:16.498425 1568 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:16.499956 1568 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:16.507241 1568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: Generated new TSK 0
I20260928 14:50:16.508095 1568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:16.511034 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.65:0
--local_ip_for_outbound_sockets=127.13.223.65
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--builtin_ntp_servers=127.13.223.84:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 14:50:16.639644 1578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:16.639942 1578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:16.639983 1578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:16.644232 1578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:16.644309 1578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:16.644367 1578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.65
I20260928 14:50:16.649497 1578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.13.223.65
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.65
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:16.650723 1578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:16.652199 1578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:16.663040 1584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:16.663030 1583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.663129 1578 server_base.cc:1034] running on GCE node
W20260928 14:50:16.663029 1586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.663812 1578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:16.664558 1578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:16.665748 1578 hybrid_clock.cc:648] HybridClock initialized: now 1790607016665698 us; error 69 us; skew 500 ppm
I20260928 14:50:16.667934 1578 webserver.cc:460] Webserver started at http://127.13.223.65:40361/ using document root <none> and password file <none>
I20260928 14:50:16.668587 1578 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:16.668656 1578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:16.668874 1578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:16.671249 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data/instance:
uuid: "4641712fcd6947ebbffcb4fd96044d79"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "b06999829699bc396b4cff1587a6c271"
server_key_iv: "bd0d105f7eefbc22237b8feaf6c761eb"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.671895 1578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance:
uuid: "4641712fcd6947ebbffcb4fd96044d79"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "b06999829699bc396b4cff1587a6c271"
server_key_iv: "bd0d105f7eefbc22237b8feaf6c761eb"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.676194 1578 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:16.679111 1593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.680794 1578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:16.681087 1578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal
uuid: "4641712fcd6947ebbffcb4fd96044d79"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "b06999829699bc396b4cff1587a6c271"
server_key_iv: "bd0d105f7eefbc22237b8feaf6c761eb"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.681242 1578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:16.701025 1578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:16.701879 1578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:16.702111 1578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:16.702935 1578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:16.704064 1578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:16.704138 1578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.704218 1578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:16.704268 1578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.714851 1578 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.65:36273
I20260928 14:50:16.714879 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.65:36273 every 8 connection(s)
I20260928 14:50:16.715946 1578 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/data/info.pb
I20260928 14:50:16.719583 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1578
I20260928 14:50:16.719728 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-0/wal/instance
I20260928 14:50:16.720067 14205 external_mini_cluster.cc:1428] Setting key 9a43b3a8bcb396134166d53fad8ce85b
I20260928 14:50:16.722635 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.66:0
--local_ip_for_outbound_sockets=127.13.223.66
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--builtin_ntp_servers=127.13.223.84:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:16.727727 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.717989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.65:34615 (local address 127.13.223.126:44245)
0928 14:50:16.718161 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.718166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.719525 (+ 1359us) server_negotiation.cc:407] Connection header received
0928 14:50:16.721437 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.721441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.721499 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.721587 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:16.723049 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.723630 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.725126 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.725545 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.725686 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.727067 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:16.727073 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:16.727079 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:16.727102 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:16.727132 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:16.727147 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:16.727266 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.727417 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.727501 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 14:50:16.728686 1710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.718485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44245 (local address 127.13.223.65:34615)
0928 14:50:16.719343 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.719393 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:16.721194 (+ 1801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:16.721815 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:16.721827 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:16.721962 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:16.722849 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.722878 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.723794 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.723801 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.724971 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.724982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.725750 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.725753 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.725933 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.726764 (+ 831us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:16.726766 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:16.726812 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:16.726889 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:16.727261 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:16.727268 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:16.727273 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 14:50:16.727372 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 14:50:16.727640 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":139,"threads_started":1}
I20260928 14:50:16.730115 1708 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44245
I20260928 14:50:16.730369 1708 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:16.730988 1708 heartbeater.cc:511] Master 127.13.223.126:44245 requested a full tablet report, sending...
I20260928 14:50:16.732820 1518 ts_manager.cc:194] Registered new tserver with Master: 4641712fcd6947ebbffcb4fd96044d79 (127.13.223.65:36273)
I20260928 14:50:16.734665 1518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.65:34615
W20260928 14:50:16.846354 1711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:16.846743 1711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:16.846789 1711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:16.851325 1711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:16.851405 1711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:16.851467 1711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.66
I20260928 14:50:16.856616 1711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.13.223.66
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.66
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:16.857789 1711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:16.859376 1711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:16.870797 1717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:16.870807 1716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.871173 1711 server_base.cc:1034] running on GCE node
W20260928 14:50:16.870810 1719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:16.871924 1711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:16.872728 1711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:16.873929 1711 hybrid_clock.cc:648] HybridClock initialized: now 1790607016873882 us; error 65 us; skew 500 ppm
I20260928 14:50:16.876283 1711 webserver.cc:460] Webserver started at http://127.13.223.66:37489/ using document root <none> and password file <none>
I20260928 14:50:16.876868 1711 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:16.876935 1711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:16.877121 1711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:16.879520 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data/instance:
uuid: "cc916315ccab4612a102e8cc97608d03"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "4fd7bb4d44f1b20a480637d776b6f6f9"
server_key_iv: "1e48b15d57de4623cc61953ea7f6fbd5"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.880194 1711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance:
uuid: "cc916315ccab4612a102e8cc97608d03"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "4fd7bb4d44f1b20a480637d776b6f6f9"
server_key_iv: "1e48b15d57de4623cc61953ea7f6fbd5"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.884377 1711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:16.887791 1725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.889537 1711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:16.889730 1711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal
uuid: "cc916315ccab4612a102e8cc97608d03"
format_stamp: "Formatted at 2026-09-28 14:50:16 on dist-test-slave-gkhx"
server_key: "4fd7bb4d44f1b20a480637d776b6f6f9"
server_key_iv: "1e48b15d57de4623cc61953ea7f6fbd5"
server_key_version: "encryptionkey@0"
I20260928 14:50:16.889878 1711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:16.923277 1711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:16.924049 1711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:16.924274 1711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:16.924917 1711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:16.925935 1711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:16.926009 1711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.926083 1711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:16.926123 1711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:16.936113 1711 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.66:32779
I20260928 14:50:16.936203 1840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.66:32779 every 8 connection(s)
I20260928 14:50:16.937134 1711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/data/info.pb
I20260928 14:50:16.940886 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1711
I20260928 14:50:16.941042 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-1/wal/instance
I20260928 14:50:16.941342 14205 external_mini_cluster.cc:1428] Setting key 65fd91676edb9820622c1dfd5c9cdcd3
I20260928 14:50:16.943423 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.13.223.67:0
--local_ip_for_outbound_sockets=127.13.223.67
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--builtin_ntp_servers=127.13.223.84:34107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 14:50:16.948928 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.939265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.66:58289 (local address 127.13.223.126:44245)
0928 14:50:16.939403 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:16.939407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:16.940678 (+ 1271us) server_negotiation.cc:407] Connection header received
0928 14:50:16.942307 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:16.942312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:16.942368 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:16.942471 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:16.943547 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.944109 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.945900 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.946209 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.946317 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.948290 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:16.948294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:16.948297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:16.948315 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:16.948339 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:16.948349 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:16.948430 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:16.948708 (+ 278us) server_negotiation.cc:299] Negotiation successful
0928 14:50:16.948794 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:50:16.950404 1843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:16.939769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44245 (local address 127.13.223.66:58289)
0928 14:50:16.940490 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:16.940537 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:16.942083 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:16.942693 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:16.942701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:16.942796 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:16.943384 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.943399 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.944258 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.944261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.945725 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:16.945741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:16.946405 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:16.946412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:16.946716 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:16.947807 (+ 1091us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:16.947812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:16.947887 (+ 75us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:16.948005 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:16.948454 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:16.948467 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:16.948477 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 14:50:16.948654 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 14:50:16.949073 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":131,"threads_started":1}
I20260928 14:50:16.951936 1841 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44245
I20260928 14:50:16.952243 1841 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:16.952987 1841 heartbeater.cc:511] Master 127.13.223.126:44245 requested a full tablet report, sending...
I20260928 14:50:16.954278 1518 ts_manager.cc:194] Registered new tserver with Master: cc916315ccab4612a102e8cc97608d03 (127.13.223.66:32779)
I20260928 14:50:16.954986 1518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.66:58289
W20260928 14:50:17.073617 1844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:17.074016 1844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:17.074110 1844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:17.078836 1844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:17.078953 1844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:17.079061 1844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.13.223.67
I20260928 14:50:17.084291 1844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.13.223.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.13.223.67
--webserver_port=0
--tserver_master_addrs=127.13.223.126:44245
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.13.223.67
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:17.085657 1844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:17.087395 1844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:17.098250 1852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:17.098249 1849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:17.098258 1850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:17.098783 1844 server_base.cc:1034] running on GCE node
I20260928 14:50:17.099308 1844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:17.100127 1844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:17.101337 1844 hybrid_clock.cc:648] HybridClock initialized: now 1790607017101293 us; error 59 us; skew 500 ppm
I20260928 14:50:17.103734 1844 webserver.cc:460] Webserver started at http://127.13.223.67:37743/ using document root <none> and password file <none>
I20260928 14:50:17.104401 1844 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:17.104475 1844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:17.104712 1844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:17.107214 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data/instance:
uuid: "ce948faff050451384dfd3da779a5e9f"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-gkhx"
server_key: "d102013ec0772e84388669805b27ae1e"
server_key_iv: "5a402c1a60f563b39b646f3068ab42cf"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.107885 1844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance:
uuid: "ce948faff050451384dfd3da779a5e9f"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-gkhx"
server_key: "d102013ec0772e84388669805b27ae1e"
server_key_iv: "5a402c1a60f563b39b646f3068ab42cf"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.113020 1844 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:50:17.116567 1858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:17.118731 1844 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:17.118909 1844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal
uuid: "ce948faff050451384dfd3da779a5e9f"
format_stamp: "Formatted at 2026-09-28 14:50:17 on dist-test-slave-gkhx"
server_key: "d102013ec0772e84388669805b27ae1e"
server_key_iv: "5a402c1a60f563b39b646f3068ab42cf"
server_key_version: "encryptionkey@0"
I20260928 14:50:17.119082 1844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:17.135226 1844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:17.136058 1844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:17.136291 1844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:17.136976 1844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 14:50:17.138072 1844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 14:50:17.138145 1844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:17.138229 1844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 14:50:17.138273 1844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:17.149019 1844 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.67:45423
I20260928 14:50:17.149107 1972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.67:45423 every 8 connection(s)
I20260928 14:50:17.150192 1844 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/data/info.pb
I20260928 14:50:17.151966 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 1844
I20260928 14:50:17.152076 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal/instance
I20260928 14:50:17.152427 14205 external_mini_cluster.cc:1428] Setting key fb282b14ea5d04ae12ac43aa710d8434
I20260928 14:50:17.161825 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.152500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:52621 (local address 127.13.223.126:44245)
0928 14:50:17.152683 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.152688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.154080 (+ 1392us) server_negotiation.cc:407] Connection header received
0928 14:50:17.155914 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.155920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.155972 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.156051 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:17.157309 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.157920 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.159465 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.159735 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.159841 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.161200 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:17.161203 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:17.161206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:17.161222 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:17.161250 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:17.161263 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:17.161346 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.161611 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.161689 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260928 14:50:17.162956 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.153059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44245 (local address 127.13.223.67:52621)
0928 14:50:17.153908 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.153973 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.155659 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.156357 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.156367 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.156473 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:17.157129 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.157146 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.158230 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.158233 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.159319 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.159331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.159904 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.159907 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.160080 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.160842 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 14:50:17.160843 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:17.160890 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 14:50:17.160969 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:17.161381 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:17.161391 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:17.161405 (+ 14us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.161542 (+ 137us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.162019 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":166,"threads_started":1}
I20260928 14:50:17.164340 1973 heartbeater.cc:347] Connected to a master server at 127.13.223.126:44245
I20260928 14:50:17.164659 1973 heartbeater.cc:464] Registering TS with master...
I20260928 14:50:17.165421 1973 heartbeater.cc:511] Master 127.13.223.126:44245 requested a full tablet report, sending...
I20260928 14:50:17.166788 1518 ts_manager.cc:194] Registered new tserver with Master: ce948faff050451384dfd3da779a5e9f (127.13.223.67:45423)
I20260928 14:50:17.167447 1518 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.13.223.67:52621
I20260928 14:50:17.179828 14205 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 14:50:17.186414 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.182199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.13.223.126:44245)
0928 14:50:17.182376 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.182381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.182590 (+ 209us) server_negotiation.cc:407] Connection header received
0928 14:50:17.182774 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.182777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.182822 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.182903 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:17.183213 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.183750 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.184670 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.184900 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.185025 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.185915 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:17.185919 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:17.185926 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:17.185939 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:17.185961 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:17.185969 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:17.186068 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.186159 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.186222 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 14:50:17.190119 1518 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49996:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 14:50:17.193810 1518 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 14:50:17.211234 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.205289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.13.223.66:32779)
0928 14:50:17.205715 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.205729 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.205785 (+ 56us) server_negotiation.cc:407] Connection header received
0928 14:50:17.206241 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.206246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.206436 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.206620 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:17.207152 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.208003 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.209651 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.210362 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.210519 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.210799 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.210912 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.211034 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":120,"threads_started":1}
I20260928 14:50:17.211781 1987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.204861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:32779 (local address 127.0.0.1:36216)
0928 14:50:17.205701 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.205715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.205807 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.206613 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.206616 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.206634 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:17.206990 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.207002 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.208101 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.208104 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.209476 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.209498 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.210513 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.210517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.210683 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.210728 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.210877 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.211042 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":780,"mutex_wait_us":57,"thread_start_us":98,"threads_started":1}
I20260928 14:50:17.212394 1986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.204489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:36273 (local address 127.0.0.1:37860)
0928 14:50:17.205064 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.205080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.205160 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.205958 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.205964 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.205997 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:17.206252 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.206263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.207811 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.207815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.209964 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.209971 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.211643 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.211647 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.211757 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.211790 (+ 33us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.212079 (+ 289us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.212192 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":1169,"spinlock_wait_cycles":6272,"thread_start_us":116,"threads_started":1}
I20260928 14:50:17.213222 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.205000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.13.223.65:36273)
0928 14:50:17.205468 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.205475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.205499 (+ 24us) server_negotiation.cc:407] Connection header received
0928 14:50:17.205581 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.205587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.205794 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.205999 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:17.206423 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.207662 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.210104 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.211039 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.211165 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.212851 (+ 1686us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.212912 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.213031 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":87,"threads_started":1}
I20260928 14:50:17.213642 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.204078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.67:45423 (local address 127.0.0.1:37946)
0928 14:50:17.204652 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.204712 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.204870 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.205948 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.205955 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.205993 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:17.206240 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.206248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.207874 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.207877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.209480 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.209495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.213179 (+ 3684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.213183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.213284 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.213309 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.213420 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.213504 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":146,"spinlock_wait_cycles":2816,"thread_start_us":93,"threads_started":1}
I20260928 14:50:17.213726 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.204374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.13.223.67:45423)
0928 14:50:17.205210 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.205225 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.205251 (+ 26us) server_negotiation.cc:407] Connection header received
0928 14:50:17.205329 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.205334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.205765 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.206135 (+ 370us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:17.206368 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.207745 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.210149 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.212998 (+ 2849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.213153 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.213352 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.213449 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.213550 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":220,"threads_started":1}
I20260928 14:50:17.215672 1774 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2ef90535aa44a7b659016b5e53679e (DEFAULT_TABLE table=test-table [id=8bb9105a96064cd0b299ba5b30821fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:17.216210 1907 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2ef90535aa44a7b659016b5e53679e (DEFAULT_TABLE table=test-table [id=8bb9105a96064cd0b299ba5b30821fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:17.216490 1642 tablet_service.cc:1467] Processing CreateTablet for tablet 1d2ef90535aa44a7b659016b5e53679e (DEFAULT_TABLE table=test-table [id=8bb9105a96064cd0b299ba5b30821fe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 14:50:17.217018 1774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2ef90535aa44a7b659016b5e53679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:17.217289 1907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2ef90535aa44a7b659016b5e53679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:17.217541 1642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d2ef90535aa44a7b659016b5e53679e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:17.224864 1990 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Bootstrap starting.
I20260928 14:50:17.226462 1992 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: Bootstrap starting.
I20260928 14:50:17.227636 1990 tablet_bootstrap.cc:654] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:17.228555 1990 log.cc:824] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:17.228864 1992 tablet_bootstrap.cc:654] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:17.229429 1991 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Bootstrap starting.
I20260928 14:50:17.229818 1992 log.cc:824] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:17.231055 1990 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: No bootstrap required, opened a new log
I20260928 14:50:17.231282 1990 ts_tablet_manager.cc:1397] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 14:50:17.232270 1992 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: No bootstrap required, opened a new log
I20260928 14:50:17.232350 1991 tablet_bootstrap.cc:654] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:17.232506 1992 ts_tablet_manager.cc:1397] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260928 14:50:17.233353 1991 log.cc:824] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:17.236126 1991 tablet_bootstrap.cc:492] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: No bootstrap required, opened a new log
I20260928 14:50:17.236399 1991 ts_tablet_manager.cc:1397] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20260928 14:50:17.236867 1990 raft_consensus.cc:348] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.237207 1990 raft_consensus.cc:374] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:17.237270 1990 raft_consensus.cc:729] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce948faff050451384dfd3da779a5e9f, State: Initialized, Role: FOLLOWER
I20260928 14:50:17.237329 1992 raft_consensus.cc:348] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.237588 1992 raft_consensus.cc:374] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:17.237655 1992 raft_consensus.cc:729] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4641712fcd6947ebbffcb4fd96044d79, State: Initialized, Role: FOLLOWER
I20260928 14:50:17.237840 1990 consensus_queue.cc:260] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.238091 1992 consensus_queue.cc:260] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.238726 1973 heartbeater.cc:503] Master 127.13.223.126:44245 was elected leader, sending a full tablet report...
I20260928 14:50:17.239060 1990 ts_tablet_manager.cc:1428] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260928 14:50:17.239060 1992 ts_tablet_manager.cc:1428] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260928 14:50:17.239559 1708 heartbeater.cc:503] Master 127.13.223.126:44245 was elected leader, sending a full tablet report...
I20260928 14:50:17.242455 1991 raft_consensus.cc:348] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.242897 1991 raft_consensus.cc:374] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:17.242986 1991 raft_consensus.cc:729] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc916315ccab4612a102e8cc97608d03, State: Initialized, Role: FOLLOWER
I20260928 14:50:17.243587 1991 consensus_queue.cc:260] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.244426 1841 heartbeater.cc:503] Master 127.13.223.126:44245 was elected leader, sending a full tablet report...
I20260928 14:50:17.246313 1991 ts_tablet_manager.cc:1428] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Time spent starting tablet: real 0.010s user 0.004s sys 0.004s
I20260928 14:50:17.263141 1996 raft_consensus.cc:482] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 14:50:17.263423 1996 raft_consensus.cc:504] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.264858 1996 leader_election.cc:288] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc916315ccab4612a102e8cc97608d03 (127.13.223.66:32779), 4641712fcd6947ebbffcb4fd96044d79 (127.13.223.65:36273)
I20260928 14:50:17.269407 1988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.265475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:57693 (local address 127.13.223.65:36273)
0928 14:50:17.265607 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.265612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.265632 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:17.265687 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.265689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.265735 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.265847 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:17.266238 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.266917 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.268212 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.268753 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.268892 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.268938 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.269161 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.269244 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:50:17.269415 1975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.265117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:36273 (local address 127.13.223.67:57693)
0928 14:50:17.265351 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.265372 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.265487 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.265864 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.265867 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.265928 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:17.266104 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.266115 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.267044 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.267048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.268023 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.268078 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.268889 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.268893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.269020 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.269051 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.269135 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.269225 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":34,"spinlock_wait_cycles":57472}
I20260928 14:50:17.269665 1989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.265421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.13.223.67:44793 (local address 127.13.223.66:32779)
0928 14:50:17.265563 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.265568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.265868 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:50:17.266075 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.266078 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.266127 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.266215 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 14:50:17.266476 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.267145 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.268383 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.269035 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.269174 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.269224 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.269431 (+ 207us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.269535 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:50:17.270054 1999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.265382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:32779 (local address 127.13.223.67:44793)
0928 14:50:17.265764 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.265775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:17.265905 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:17.266226 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:17.266229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:17.266245 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 14:50:17.266362 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.266367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.267263 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.267269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.268248 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:17.268258 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.269173 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.269177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:17.269276 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.269298 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 14:50:17.269800 (+ 502us) client_negotiation.cc:240] Negotiation successful
0928 14:50:17.269902 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"mutex_wait_us":28,"spinlock_wait_cycles":3968,"thread_start_us":137,"threads_started":1}
I20260928 14:50:17.270249 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2ef90535aa44a7b659016b5e53679e" candidate_uuid: "ce948faff050451384dfd3da779a5e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4641712fcd6947ebbffcb4fd96044d79" is_pre_election: true
I20260928 14:50:17.270648 1662 raft_consensus.cc:2454] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce948faff050451384dfd3da779a5e9f in term 0.
I20260928 14:50:17.270892 1795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2ef90535aa44a7b659016b5e53679e" candidate_uuid: "ce948faff050451384dfd3da779a5e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc916315ccab4612a102e8cc97608d03" is_pre_election: true
I20260928 14:50:17.271190 1860 leader_election.cc:302] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4641712fcd6947ebbffcb4fd96044d79, ce948faff050451384dfd3da779a5e9f; no voters:
I20260928 14:50:17.271320 1795 raft_consensus.cc:2454] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce948faff050451384dfd3da779a5e9f in term 0.
I20260928 14:50:17.271462 1996 raft_consensus.cc:2781] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 14:50:17.271545 1996 raft_consensus.cc:482] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 14:50:17.271606 1996 raft_consensus.cc:3037] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:17.272728 1996 raft_consensus.cc:504] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.273092 1996 leader_election.cc:288] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [CANDIDATE]: Term 1 election: Requested vote from peers cc916315ccab4612a102e8cc97608d03 (127.13.223.66:32779), 4641712fcd6947ebbffcb4fd96044d79 (127.13.223.65:36273)
I20260928 14:50:17.273486 1795 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2ef90535aa44a7b659016b5e53679e" candidate_uuid: "ce948faff050451384dfd3da779a5e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc916315ccab4612a102e8cc97608d03"
I20260928 14:50:17.273470 1662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d2ef90535aa44a7b659016b5e53679e" candidate_uuid: "ce948faff050451384dfd3da779a5e9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4641712fcd6947ebbffcb4fd96044d79"
I20260928 14:50:17.273605 1795 raft_consensus.cc:3037] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:17.273751 1662 raft_consensus.cc:3037] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:17.275046 1662 raft_consensus.cc:2454] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce948faff050451384dfd3da779a5e9f in term 1.
I20260928 14:50:17.275063 1795 raft_consensus.cc:2454] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce948faff050451384dfd3da779a5e9f in term 1.
I20260928 14:50:17.275523 1861 leader_election.cc:302] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4641712fcd6947ebbffcb4fd96044d79, cc916315ccab4612a102e8cc97608d03, ce948faff050451384dfd3da779a5e9f; no voters:
I20260928 14:50:17.275754 1996 raft_consensus.cc:2781] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:17.276068 1996 raft_consensus.cc:686] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 1 LEADER]: Becoming Leader. State: Replica: ce948faff050451384dfd3da779a5e9f, State: Running, Role: LEADER
I20260928 14:50:17.276472 1996 consensus_queue.cc:237] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } }
I20260928 14:50:17.279947 1518 catalog_manager.cc:5495] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f reported cstate change: term changed from 0 to 1, leader changed from <none> to ce948faff050451384dfd3da779a5e9f (127.13.223.67). New cstate: current_term: 1 leader_uuid: "ce948faff050451384dfd3da779a5e9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce948faff050451384dfd3da779a5e9f" member_type: VOTER last_known_addr { host: "127.13.223.67" port: 45423 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 } health_report { overall_health: UNKNOWN } } }
I20260928 14:50:17.295102 1985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.291058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37950 (local address 127.13.223.67:45423)
0928 14:50:17.291235 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:17.291240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:17.291339 (+ 99us) server_negotiation.cc:407] Connection header received
0928 14:50:17.291565 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:17.291568 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:17.291631 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:17.291749 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:17.291992 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.292556 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.293434 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:17.293721 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:17.293862 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:17.294176 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:17.294704 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:17.294834 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:17.294881 (+ 47us) server_negotiation.cc:299] Negotiation successful
0928 14:50:17.294944 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 14:50:17.301900 1662 raft_consensus.cc:1261] T 1d2ef90535aa44a7b659016b5e53679e P 4641712fcd6947ebbffcb4fd96044d79 [term 1 FOLLOWER]: Refusing update from remote peer ce948faff050451384dfd3da779a5e9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:17.301918 1795 raft_consensus.cc:1261] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 1 FOLLOWER]: Refusing update from remote peer ce948faff050451384dfd3da779a5e9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:17.302776 2000 consensus_queue.cc:1035] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc916315ccab4612a102e8cc97608d03" member_type: VOTER last_known_addr { host: "127.13.223.66" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:17.303048 1996 consensus_queue.cc:1035] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4641712fcd6947ebbffcb4fd96044d79" member_type: VOTER last_known_addr { host: "127.13.223.65" port: 36273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:17.310848 2003 mvcc.cc:205] Tried to move back new op lower bound from 7334326342859972608 to 7334326342767521792. Current Snapshot: MvccSnapshot[applied={T|T < 7334326342859972608}]
I20260928 14:50:17.312781 2004 mvcc.cc:205] Tried to move back new op lower bound from 7334326342859972608 to 7334326342767521792. Current Snapshot: MvccSnapshot[applied={T|T < 7334326342859972608}]
I20260928 14:50:17.313898 2005 mvcc.cc:205] Tried to move back new op lower bound from 7334326342859972608 to 7334326342767521792. Current Snapshot: MvccSnapshot[applied={T|T < 7334326342859972608}]
I20260928 14:50:17.322293 1516 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:17.322535 1516 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49996:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 14:50:17.325786 1516 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f398fa0c30f740e9b5e6f0b58b449ec4: Sending DeleteTablet for 3 replicas of tablet 1d2ef90535aa44a7b659016b5e53679e
I20260928 14:50:17.327039 1774 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2ef90535aa44a7b659016b5e53679e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:17 UTC) from {username='slave'} at 127.0.0.1:36216
I20260928 14:50:17.327044 1642 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2ef90535aa44a7b659016b5e53679e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:17 UTC) from {username='slave'} at 127.0.0.1:37860
I20260928 14:50:17.327147 1907 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d2ef90535aa44a7b659016b5e53679e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 14:50:17 UTC) from {username='slave'} at 127.0.0.1:37946
I20260928 14:50:17.327991 2015 tablet_replica.cc:330] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: stopping tablet replica
I20260928 14:50:17.328289 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1578
I20260928 14:50:17.328378 2015 raft_consensus.cc:2229] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 1 LEADER]: Raft consensus shutting down.
I20260928 14:50:17.328948 2015 raft_consensus.cc:2258] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:17.330849 2015 ts_tablet_manager.cc:1905] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 14:50:17.334249 2015 ts_tablet_manager.cc:1918] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:17.334375 2015 log.cc:1196] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Deleting WAL directory at /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790606739990168-14205-0/minicluster-data/ts-2/wal/wals/1d2ef90535aa44a7b659016b5e53679e
I20260928 14:50:17.334663 2014 tablet_replica.cc:330] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: stopping tablet replica
I20260928 14:50:17.334829 2015 ts_tablet_manager.cc:1939] T 1d2ef90535aa44a7b659016b5e53679e P ce948faff050451384dfd3da779a5e9f: Deleting consensus metadata
I20260928 14:50:17.335206 2014 raft_consensus.cc:2229] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 14:50:17.335630 2014 raft_consensus.cc:2258] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 14:50:17.336305 1504 catalog_manager.cc:4841] TS ce948faff050451384dfd3da779a5e9f (127.13.223.67:45423): tablet 1d2ef90535aa44a7b659016b5e53679e (table test-table [id=8bb9105a96064cd0b299ba5b30821fe5]) successfully deleted
I20260928 14:50:17.337805 2014 ts_tablet_manager.cc:1905] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 14:50:17.341148 1503 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.13.223.65:36273 (error 108)
I20260928 14:50:17.341958 2014 ts_tablet_manager.cc:1918] T 1d2ef90535aa44a7b659016b5e53679e P cc916315ccab4612a102e8cc97608d03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 14:50:17.342190 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1711
I20260928 14:50:17.343112 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.342244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.65:36273 (local address 127.0.0.1:37862)
0928 14:50:17.342889 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.342991 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.65:36273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500}
W20260928 14:50:17.343451 1503 catalog_manager.cc:4584] TS 4641712fcd6947ebbffcb4fd96044d79 (127.13.223.65:36273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d2ef90535aa44a7b659016b5e53679e: Network error: Client connection negotiation failed: client connection to 127.13.223.65:36273: connect: Connection refused (error 111)
I20260928 14:50:17.349160 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1844
I20260928 14:50:17.349519 1984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:17.349147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.66:32779 (local address 127.0.0.1:36232)
0928 14:50:17.349332 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:17.349405 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.66:32779: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 14:50:17.355690 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 1487
2026-09-28T14:50:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1180 ms)
[----------] 6 tests from EncryptionPolicyTest (11481 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:17 dist-test-slave-gkhx krb5kdc[2019](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:17 dist-test-slave-gkhx krb5kdc[2019](info): set up 2 sockets
Sep 28 14:50:17 dist-test-slave-gkhx krb5kdc[2019](info): commencing operation
krb5kdc: starting...
W20260928 14:50:18.436858 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:18.578644 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:18.702519 2032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:18.702935 2032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:18.703104 2032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:18.707564 2032 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:18.707672 2032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:18.707708 2032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:18.707747 2032 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:18.707789 2032 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:18.713853 2032 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43281
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:18.715210 2032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:18.716344 2032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:18.725795 2040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.725780 2038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:18.725780 2037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:18.726359 2032 server_base.cc:1034] running on GCE node
I20260928 14:50:18.727138 2032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:18.728413 2032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:18.729637 2032 hybrid_clock.cc:648] HybridClock initialized: now 1790607018729629 us; error 38 us; skew 500 ppm
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2019](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:18.732816 2032 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:18.733987 2032 webserver.cc:460] Webserver started at http://127.0.0.1:43599/ using document root <none> and password file <none>
I20260928 14:50:18.734687 2032 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:18.734792 2032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:18.735052 2032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:18.737298 2032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "bdfb5928f1144cb981541dd856f584ef"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-gkhx"
server_key: "2b224b99daeec815fcbb09f370780b43"
server_key_iv: "0f28397b6eadbd2c4698acaba999463d"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.737938 2032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "bdfb5928f1144cb981541dd856f584ef"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-gkhx"
server_key: "2b224b99daeec815fcbb09f370780b43"
server_key_iv: "0f28397b6eadbd2c4698acaba999463d"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.742285 2032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:18.745327 2047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:18.747022 2032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:18.747196 2032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "bdfb5928f1144cb981541dd856f584ef"
format_stamp: "Formatted at 2026-09-28 14:50:18 on dist-test-slave-gkhx"
server_key: "2b224b99daeec815fcbb09f370780b43"
server_key_iv: "0f28397b6eadbd2c4698acaba999463d"
server_key_version: "encryptionkey@0"
I20260928 14:50:18.747334 2032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:18.774972 2032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:18.778951 2032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:18.779197 2032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:18.788303 2032 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43281
I20260928 14:50:18.788335 2100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43281 every 8 connection(s)
I20260928 14:50:18.789482 2032 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:18.792852 2101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:18.797262 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2032
I20260928 14:50:18.797427 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:18.797791 14205 external_mini_cluster.cc:1428] Setting key 010861b3f0c4e23fd69123d95a522169
I20260928 14:50:18.799970 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: Bootstrap starting.
I20260928 14:50:18.802390 2101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:18.803292 2101 log.cc:824] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:18.805713 2101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: No bootstrap required, opened a new log
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2019](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:50:18.811331 2101 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260928 14:50:18.811705 2101 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:18.811762 2101 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdfb5928f1144cb981541dd856f584ef, State: Initialized, Role: FOLLOWER
I20260928 14:50:18.812382 2101 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260928 14:50:18.812587 2101 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:18.812672 2101 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:18.812801 2101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:18.813882 2101 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260928 14:50:18.814363 2101 leader_election.cc:302] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdfb5928f1144cb981541dd856f584ef; no voters:
I20260928 14:50:18.814810 2101 leader_election.cc:288] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:18.814895 2106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:18.815114 2106 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [term 1 LEADER]: Becoming Leader. State: Replica: bdfb5928f1144cb981541dd856f584ef, State: Running, Role: LEADER
I20260928 14:50:18.815399 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.799486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34923 (local address 127.0.0.1:43281)
0928 14:50:18.799998 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:18.800018 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:18.800066 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:50:18.800839 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:18.800860 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:18.801237 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:18.801616 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:18.802470 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.803686 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.804910 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.805468 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.805603 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.809017 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:18.809027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:18.809058 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:18.809113 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:18.811734 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.812417 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.812424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.812431 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.812617 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.813062 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.813065 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.813066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.813510 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:18.813754 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:18.814102 (+ 348us) server_negotiation.cc:299] Negotiation successful
0928 14:50:18.814354 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":155,"threads_started":1}
I20260928 14:50:18.815447 2106 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260928 14:50:18.816296 2101 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:18.818773 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:18.818835 2108 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdfb5928f1144cb981541dd856f584ef. Latest consensus state: current_term: 1 leader_uuid: "bdfb5928f1144cb981541dd856f584ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } } }
I20260928 14:50:18.819072 2107 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdfb5928f1144cb981541dd856f584ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdfb5928f1144cb981541dd856f584ef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } } }
I20260928 14:50:18.819223 2108 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:18.819223 2107 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef [sys.catalog]: This master's current role is: LEADER
W20260928 14:50:18.822113 2121 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:18.822228 2121 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:18.822491 2122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:18.823565 2122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:18.830667 2122 catalog_manager.cc:1347] Generated new cluster ID: e366cd3f409a47c491942fbadde0285b
I20260928 14:50:18.830742 2122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:18.832448 2104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:18.821075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34717 (local address 127.0.0.1:43281)
0928 14:50:18.821191 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:18.821195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:18.822967 (+ 1772us) server_negotiation.cc:407] Connection header received
0928 14:50:18.823123 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:18.823126 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:18.823191 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:18.823294 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:18.824207 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.825052 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.826374 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:18.826783 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:18.826911 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:18.828739 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:18.828743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:18.828746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:18.828778 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:18.830316 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.830945 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.830948 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.830950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.831058 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:18.831451 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:18.831455 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:18.831457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:18.831701 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:18.831860 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:18.832163 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 14:50:18.832276 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 14:50:18.842541 2122 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:18.843654 2122 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:18.862725 2122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdfb5928f1144cb981541dd856f584ef: Generated new TSK 0
I20260928 14:50:18.863565 2122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:18.872238 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2032
W20260928 14:50:18.878821 2017 connection.cc:535] client connection to 127.0.0.1:43281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43281): Connection reset by peer (error 104)
2026-09-28T14:50:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2133](info): set up 2 sockets
Sep 28 14:50:18 dist-test-slave-gkhx krb5kdc[2133](info): commencing operation
krb5kdc: starting...
W20260928 14:50:19.981781 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:20 dist-test-slave-gkhx krb5kdc[2133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:20.144333 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:20.272352 2147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:20.272760 2147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:20.272842 2147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:20.278296 2147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:20.278425 2147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:20.278461 2147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:20.278505 2147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:20.278542 2147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:20.284816 2147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42051
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:20.286156 2147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:20.287325 2147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:20.297101 2155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.297036 2152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:20.297046 2153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:20.297225 2147 server_base.cc:1034] running on GCE node
I20260928 14:50:20.298043 2147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:20.299363 2147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:20.300601 2147 hybrid_clock.cc:648] HybridClock initialized: now 1790607020300575 us; error 40 us; skew 500 ppm
Sep 28 14:50:20 dist-test-slave-gkhx krb5kdc[2133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607020, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:20.303794 2147 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:20.304736 2147 webserver.cc:460] Webserver started at http://127.0.0.1:33941/ using document root <none> and password file <none>
I20260928 14:50:20.305290 2147 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:20.305352 2147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:20.305527 2147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:20.307749 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "bfd07b99ee6949b0917bf81078e8b8b1"
format_stamp: "Formatted at 2026-09-28 14:50:20 on dist-test-slave-gkhx"
server_key: "431bb1808758c4b14802f7a8f16af22e"
server_key_iv: "2b9f07595d0c6d05f84766666d73a888"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.308332 2147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "bfd07b99ee6949b0917bf81078e8b8b1"
format_stamp: "Formatted at 2026-09-28 14:50:20 on dist-test-slave-gkhx"
server_key: "431bb1808758c4b14802f7a8f16af22e"
server_key_iv: "2b9f07595d0c6d05f84766666d73a888"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.312527 2147 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:20.315407 2162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:20.316864 2147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:20.316977 2147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "bfd07b99ee6949b0917bf81078e8b8b1"
format_stamp: "Formatted at 2026-09-28 14:50:20 on dist-test-slave-gkhx"
server_key: "431bb1808758c4b14802f7a8f16af22e"
server_key_iv: "2b9f07595d0c6d05f84766666d73a888"
server_key_version: "encryptionkey@0"
I20260928 14:50:20.317070 2147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:20.340214 2147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:20.344165 2147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:20.344426 2147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:20.353344 2147 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42051
I20260928 14:50:20.353353 2215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42051 every 8 connection(s)
I20260928 14:50:20.354571 2147 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:20.358003 2216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:20.362792 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2147
I20260928 14:50:20.362970 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:20.363325 14205 external_mini_cluster.cc:1428] Setting key 69319baaad72ee9b6228dd82db40d804
I20260928 14:50:20.365438 2216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1: Bootstrap starting.
I20260928 14:50:20.368063 2216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:20.368927 2216 log.cc:824] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:20.371400 2216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1: No bootstrap required, opened a new log
I20260928 14:50:20.373634 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.364959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33025 (local address 127.0.0.1:42051)
0928 14:50:20.365461 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:20.365533 (+ 72us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:20.365587 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:50:20.366316 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:20.366337 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:20.366795 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:20.367161 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:20.367616 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.368758 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.369741 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.370242 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.370419 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.371440 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:20.371445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:20.371459 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:20.371926 (+ 467us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:20.371986 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:20.372012 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:20.372236 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:20.372416 (+ 180us) server_negotiation.cc:299] Negotiation successful
0928 14:50:20.372572 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":148,"threads_started":1}
I20260928 14:50:20.375250 2216 raft_consensus.cc:348] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260928 14:50:20.375590 2216 raft_consensus.cc:374] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:20.375646 2216 raft_consensus.cc:729] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfd07b99ee6949b0917bf81078e8b8b1, State: Initialized, Role: FOLLOWER
I20260928 14:50:20.376413 2216 consensus_queue.cc:260] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260928 14:50:20.376610 2216 raft_consensus.cc:388] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:20.376706 2216 raft_consensus.cc:482] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:20.376819 2216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:20.377295 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:20.378091 2216 raft_consensus.cc:504] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260928 14:50:20.378526 2216 leader_election.cc:302] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bfd07b99ee6949b0917bf81078e8b8b1; no voters:
I20260928 14:50:20.378991 2216 leader_election.cc:288] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:20.379052 2225 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:20.379261 2225 raft_consensus.cc:686] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [term 1 LEADER]: Becoming Leader. State: Replica: bfd07b99ee6949b0917bf81078e8b8b1, State: Running, Role: LEADER
I20260928 14:50:20.379635 2225 consensus_queue.cc:237] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } }
I20260928 14:50:20.380518 2216 sys_catalog.cc:564] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:20.380988 2225 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260928 14:50:20.381145 2225 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:20.380966 2227 sys_catalog.cc:455] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bfd07b99ee6949b0917bf81078e8b8b1. Latest consensus state: current_term: 1 leader_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bfd07b99ee6949b0917bf81078e8b8b1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42051 } } }
I20260928 14:50:20.381371 2227 sys_catalog.cc:458] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:20.381711 2230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:20.386006 2230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:20.386664 2219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:20.379608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59981 (local address 127.0.0.1:42051)
0928 14:50:20.379814 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:20.379819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:20.381572 (+ 1753us) server_negotiation.cc:407] Connection header received
0928 14:50:20.381805 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:20.381809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:20.381866 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:20.381943 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:20.382426 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.383254 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.384473 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:20.384711 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:20.384827 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:20.385787 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:20.385790 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:20.385793 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:20.385811 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:20.385837 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:20.385848 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:20.385943 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:20.386454 (+ 511us) server_negotiation.cc:299] Negotiation successful
0928 14:50:20.386489 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 14:50:20.393216 2230 catalog_manager.cc:1347] Generated new cluster ID: f1d58e04394e4c9f8a62d779267311ba
I20260928 14:50:20.393325 2230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:20.427385 2230 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:20.428613 2230 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:20.434481 2230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bfd07b99ee6949b0917bf81078e8b8b1: Generated new TSK 0
I20260928 14:50:20.435519 2230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:20.445736 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2147
W20260928 14:50:20.452049 2131 connection.cc:535] client connection to 127.0.0.1:42051 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42051): Connection reset by peer (error 104)
2026-09-28T14:50:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1573 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:20 dist-test-slave-gkhx krb5kdc[2247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:20 dist-test-slave-gkhx krb5kdc[2247](info): set up 2 sockets
Sep 28 14:50:20 dist-test-slave-gkhx krb5kdc[2247](info): commencing operation
krb5kdc: starting...
W20260928 14:50:21.561559 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:21 dist-test-slave-gkhx krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:21.706527 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34395
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:21.824922 2260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:21.825274 2260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:21.825345 2260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:21.829605 2260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:21.829718 2260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:21.829758 2260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:21.829805 2260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:21.829840 2260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:21.835672 2260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34395
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:21.836860 2260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:21.838007 2260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:50:21.847235 2260 server_base.cc:1034] running on GCE node
W20260928 14:50:21.847239 2267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:21.847347 2266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:21.847240 2269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:21.848199 2260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:21.849387 2260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:21.850625 2260 hybrid_clock.cc:648] HybridClock initialized: now 1790607021850586 us; error 54 us; skew 500 ppm
Sep 28 14:50:21 dist-test-slave-gkhx krb5kdc[2247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607021, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:21.859230 2260 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:21.860337 2260 webserver.cc:460] Webserver started at http://127.0.0.1:46123/ using document root <none> and password file <none>
I20260928 14:50:21.860985 2260 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:21.861083 2260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:21.861323 2260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:21.863622 2260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "97ee3d4e4f1e451c8d060c649e40d207"
format_stamp: "Formatted at 2026-09-28 14:50:21 on dist-test-slave-gkhx"
server_key: "9ae03626fcf667ba2c47e8f120e90c1e"
server_key_iv: "1ff0b6a38666e8d8d7b7ca43c32ecad7"
server_key_version: "encryptionkey@0"
I20260928 14:50:21.864246 2260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "97ee3d4e4f1e451c8d060c649e40d207"
format_stamp: "Formatted at 2026-09-28 14:50:21 on dist-test-slave-gkhx"
server_key: "9ae03626fcf667ba2c47e8f120e90c1e"
server_key_iv: "1ff0b6a38666e8d8d7b7ca43c32ecad7"
server_key_version: "encryptionkey@0"
I20260928 14:50:21.868450 2260 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:21.871747 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:21.873574 2260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:21.873797 2260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "97ee3d4e4f1e451c8d060c649e40d207"
format_stamp: "Formatted at 2026-09-28 14:50:21 on dist-test-slave-gkhx"
server_key: "9ae03626fcf667ba2c47e8f120e90c1e"
server_key_iv: "1ff0b6a38666e8d8d7b7ca43c32ecad7"
server_key_version: "encryptionkey@0"
I20260928 14:50:21.873968 2260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:21.879747 2260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:21.883744 2260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:21.883998 2260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:21.892778 2260 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34395
I20260928 14:50:21.893302 2328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34395 every 8 connection(s)
I20260928 14:50:21.894003 2260 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:21.894344 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2260
I20260928 14:50:21.894536 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:21.894912 14205 external_mini_cluster.cc:1428] Setting key b0ca1c0cd6dc4d90066dc2db0ac32634
I20260928 14:50:21.897586 2329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:21.901070 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:21.896513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58105 (local address 127.0.0.1:34395)
0928 14:50:21.897057 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:21.897079 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:21.897121 (+ 42us) server_negotiation.cc:407] Connection header received
0928 14:50:21.897928 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:21.897948 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:21.898280 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:21.898737 (+ 457us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:21.899626 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:21.899631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:21.899645 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:21.899656 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:21.899717 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:21.899739 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:21.899908 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:21.899991 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 14:50:21.900173 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":162,"threads_started":1}
I20260928 14:50:21.904536 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:21.904976 2329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207: Bootstrap starting.
I20260928 14:50:21.907778 2329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:21.908802 2329 log.cc:824] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:21.909111 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:21.907188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41311 (local address 127.0.0.1:34395)
0928 14:50:21.907334 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:21.907340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:21.907469 (+ 129us) server_negotiation.cc:407] Connection header received
0928 14:50:21.907677 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:21.907681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:21.907730 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:21.907823 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:21.908667 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:21.908670 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:21.908673 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:21.908685 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:21.908727 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:21.908745 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:21.908844 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:21.908925 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 14:50:21.908973 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:50:21.911609 2329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207: No bootstrap required, opened a new log
I20260928 14:50:21.916198 2329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260928 14:50:21.916616 2329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:21.916705 2329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97ee3d4e4f1e451c8d060c649e40d207, State: Initialized, Role: FOLLOWER
I20260928 14:50:21.917271 2329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260928 14:50:21.917486 2329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:21.917572 2329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:21.917699 2329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:21.919474 2329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260928 14:50:21.919906 2329 leader_election.cc:302] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97ee3d4e4f1e451c8d060c649e40d207; no voters:
I20260928 14:50:21.920341 2329 leader_election.cc:288] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:21.920498 2339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:21.920761 2339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [term 1 LEADER]: Becoming Leader. State: Replica: 97ee3d4e4f1e451c8d060c649e40d207, State: Running, Role: LEADER
I20260928 14:50:21.921270 2339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } }
I20260928 14:50:21.922048 2329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:21.923296 2341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97ee3d4e4f1e451c8d060c649e40d207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } } }
I20260928 14:50:21.923331 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97ee3d4e4f1e451c8d060c649e40d207. Latest consensus state: current_term: 1 leader_uuid: "97ee3d4e4f1e451c8d060c649e40d207" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97ee3d4e4f1e451c8d060c649e40d207" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34395 } } }
I20260928 14:50:21.923466 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:21.923466 2341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:21.924054 2348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:21.927268 2348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:21.933729 2348 catalog_manager.cc:1347] Generated new cluster ID: 7be5388afac8408f9c622bad4ba689a5
I20260928 14:50:21.933830 2348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:21.948853 2348 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:21.949297 2348 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:21.954200 2348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97ee3d4e4f1e451c8d060c649e40d207: Generated new TSK 0
I20260928 14:50:21.955111 2348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:21.959244 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2260
2026-09-28T14:50:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:22 dist-test-slave-gkhx krb5kdc[2360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:22 dist-test-slave-gkhx krb5kdc[2360](info): set up 2 sockets
Sep 28 14:50:22 dist-test-slave-gkhx krb5kdc[2360](info): commencing operation
krb5kdc: starting...
W20260928 14:50:23.065670 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:23.207829 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32975
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39611
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:23.330721 2373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:23.331163 2373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:23.331257 2373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:23.336081 2373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:23.336213 2373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:23.336251 2373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:23.336294 2373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:23.336329 2373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:23.342162 2373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39611
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:23.343513 2373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:23.344657 2373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:23.354331 2378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:23.354331 2381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:23.354349 2379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:23.354866 2373 server_base.cc:1034] running on GCE node
I20260928 14:50:23.355664 2373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:23.357035 2373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:23.358279 2373 hybrid_clock.cc:648] HybridClock initialized: now 1790607023358256 us; error 43 us; skew 500 ppm
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607023, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:23.361722 2373 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:23.363013 2373 webserver.cc:460] Webserver started at http://127.0.0.1:34823/ using document root <none> and password file <none>
I20260928 14:50:23.363762 2373 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:23.363881 2373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:23.364158 2373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:23.366671 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "89c8e124869642cf9d61d6eb3539c352"
format_stamp: "Formatted at 2026-09-28 14:50:23 on dist-test-slave-gkhx"
server_key: "d5bd5e88e9a3ccc90827d8fee2e8050c"
server_key_iv: "1660c5f850b580cd722291340cac8809"
server_key_version: "encryptionkey@0"
I20260928 14:50:23.367434 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "89c8e124869642cf9d61d6eb3539c352"
format_stamp: "Formatted at 2026-09-28 14:50:23 on dist-test-slave-gkhx"
server_key: "d5bd5e88e9a3ccc90827d8fee2e8050c"
server_key_iv: "1660c5f850b580cd722291340cac8809"
server_key_version: "encryptionkey@0"
I20260928 14:50:23.372007 2373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 14:50:23.375074 2388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:23.376798 2373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:23.376940 2373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "89c8e124869642cf9d61d6eb3539c352"
format_stamp: "Formatted at 2026-09-28 14:50:23 on dist-test-slave-gkhx"
server_key: "d5bd5e88e9a3ccc90827d8fee2e8050c"
server_key_iv: "1660c5f850b580cd722291340cac8809"
server_key_version: "encryptionkey@0"
I20260928 14:50:23.377071 2373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:23.398003 2373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:23.401887 2373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:23.402100 2373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:23.410657 2373 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32975
I20260928 14:50:23.410642 2442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32975 every 8 connection(s)
I20260928 14:50:23.411768 2373 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:23.415002 2443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:23.416217 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2373
I20260928 14:50:23.416374 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:23.416693 14205 external_mini_cluster.cc:1428] Setting key ff9774a2c389e6e3220df2d4c8c22f26
I20260928 14:50:23.421829 2443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: Bootstrap starting.
I20260928 14:50:23.424695 2443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:23.425673 2443 log.cc:824] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607023, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:50:23.428498 2443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: No bootstrap required, opened a new log
I20260928 14:50:23.432225 2443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } }
I20260928 14:50:23.432528 2443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:23.432587 2443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89c8e124869642cf9d61d6eb3539c352, State: Initialized, Role: FOLLOWER
I20260928 14:50:23.433068 2443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } }
I20260928 14:50:23.433246 2443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:23.433334 2443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:23.433467 2443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:23.433751 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:23.418359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54141 (local address 127.0.0.1:32975)
0928 14:50:23.418938 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:23.418955 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:23.418995 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:50:23.419872 (+ 877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:23.419895 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:23.420359 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:23.420822 (+ 463us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:23.421867 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:23.423239 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:23.424497 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:23.425163 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:23.425303 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:23.428537 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:23.428543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:23.428558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:23.428589 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:23.430434 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:23.430977 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:23.430983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:23.430989 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:23.431131 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:23.431430 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:23.431435 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:23.431436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:23.431882 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:23.432126 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:23.432366 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 14:50:23.432713 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":210,"threads_started":1}
I20260928 14:50:23.434845 2443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } }
I20260928 14:50:23.435235 2443 leader_election.cc:302] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89c8e124869642cf9d61d6eb3539c352; no voters:
I20260928 14:50:23.435611 2443 leader_election.cc:288] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:23.435699 2448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:23.435883 2448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [term 1 LEADER]: Becoming Leader. State: Replica: 89c8e124869642cf9d61d6eb3539c352, State: Running, Role: LEADER
I20260928 14:50:23.436539 2448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } }
I20260928 14:50:23.437001 2443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:23.437299 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:23.438344 2450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89c8e124869642cf9d61d6eb3539c352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } } }
I20260928 14:50:23.438493 2450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:23.438868 2449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89c8e124869642cf9d61d6eb3539c352. Latest consensus state: current_term: 1 leader_uuid: "89c8e124869642cf9d61d6eb3539c352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89c8e124869642cf9d61d6eb3539c352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32975 } } }
I20260928 14:50:23.438984 2449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352 [sys.catalog]: This master's current role is: LEADER
W20260928 14:50:23.442652 2467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:23.442744 2467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:23.442803 2460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:23.443625 2460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:23.450419 2460 catalog_manager.cc:1347] Generated new cluster ID: 7a92b0c987d44b1ebc00b12bfaf28559
I20260928 14:50:23.450503 2460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:23.451757 2446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:23.440289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58551 (local address 127.0.0.1:32975)
0928 14:50:23.440418 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:23.440423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:23.441221 (+ 798us) server_negotiation.cc:407] Connection header received
0928 14:50:23.441268 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:23.441271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:23.441320 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:23.441402 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:23.442345 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:23.443133 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:23.444359 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:23.444777 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:23.444893 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:23.447057 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:23.447063 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:23.447068 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:23.447104 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:23.449558 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:23.450182 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:23.450186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:23.450188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:23.450289 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:23.450793 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:23.450799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:23.450801 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:23.451011 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:23.451171 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:23.451445 (+ 274us) server_negotiation.cc:299] Negotiation successful
0928 14:50:23.451579 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 14:50:23.460767 2460 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:23.461975 2460 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:23.472347 2460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89c8e124869642cf9d61d6eb3539c352: Generated new TSK 0
I20260928 14:50:23.473150 2460 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:23.480422 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2373
2026-09-28T14:50:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1521 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2474](info): set up 2 sockets
Sep 28 14:50:23 dist-test-slave-gkhx krb5kdc[2474](info): commencing operation
krb5kdc: starting...
W20260928 14:50:24.557749 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:24 dist-test-slave-gkhx krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:24.703622 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:24.828553 2487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:24.829079 2487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:24.829203 2487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:24.833597 2487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:24.833702 2487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:24.833745 2487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:24.833801 2487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:24.833839 2487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:24.839850 2487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:24.841120 2487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:24.842233 2487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:24.851469 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:24.851500 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:24.851469 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:24.852197 2487 server_base.cc:1034] running on GCE node
I20260928 14:50:24.852846 2487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:24.854071 2487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:24.855297 2487 hybrid_clock.cc:648] HybridClock initialized: now 1790607024855180 us; error 132 us; skew 500 ppm
Sep 28 14:50:24 dist-test-slave-gkhx krb5kdc[2474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607024, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:24.858678 2487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:24.859817 2487 webserver.cc:460] Webserver started at http://127.0.0.1:38945/ using document root <none> and password file <none>
I20260928 14:50:24.860503 2487 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:24.860615 2487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:24.860931 2487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:24.863505 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "fd9614d5590845a195b41d0c8c4cb0ff"
format_stamp: "Formatted at 2026-09-28 14:50:24 on dist-test-slave-gkhx"
server_key: "b045300b70502e39b0159893b1c55d73"
server_key_iv: "0e47918d1ba88b2b66c27c91a6eda7fe"
server_key_version: "encryptionkey@0"
I20260928 14:50:24.864213 2487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "fd9614d5590845a195b41d0c8c4cb0ff"
format_stamp: "Formatted at 2026-09-28 14:50:24 on dist-test-slave-gkhx"
server_key: "b045300b70502e39b0159893b1c55d73"
server_key_iv: "0e47918d1ba88b2b66c27c91a6eda7fe"
server_key_version: "encryptionkey@0"
I20260928 14:50:24.868880 2487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:24.872007 2502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:24.873454 2487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 14:50:24.873646 2487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "fd9614d5590845a195b41d0c8c4cb0ff"
format_stamp: "Formatted at 2026-09-28 14:50:24 on dist-test-slave-gkhx"
server_key: "b045300b70502e39b0159893b1c55d73"
server_key_iv: "0e47918d1ba88b2b66c27c91a6eda7fe"
server_key_version: "encryptionkey@0"
I20260928 14:50:24.873783 2487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:24.893425 2487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:24.897009 2487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:24.897236 2487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:24.905246 2487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33163
I20260928 14:50:24.905272 2556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33163 every 8 connection(s)
I20260928 14:50:24.906292 2487 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:24.909338 2557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:24.911950 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2487
I20260928 14:50:24.912084 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:24.912521 14205 external_mini_cluster.cc:1428] Setting key 9a6f1a215a7a04139a3fb2b99bef7759
I20260928 14:50:24.916417 2557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff: Bootstrap starting.
I20260928 14:50:24.919360 2557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:24.920380 2557 log.cc:824] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:24.923372 2557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff: No bootstrap required, opened a new log
I20260928 14:50:24.924230 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:24.914156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35597 (local address 127.0.0.1:33163)
0928 14:50:24.914668 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:24.914687 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:24.914733 (+ 46us) server_negotiation.cc:407] Connection header received
0928 14:50:24.915596 (+ 863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:24.915622 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:24.916089 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:24.916479 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:24.916938 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:24.918529 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:24.919748 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:24.920526 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:24.920715 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:24.921733 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:24.921741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:24.921760 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:24.922411 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:24.922498 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:24.922534 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:24.922693 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:24.922903 (+ 210us) server_negotiation.cc:299] Negotiation successful
0928 14:50:24.923135 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":132,"threads_started":1}
I20260928 14:50:24.927672 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:24.928273 2557 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260928 14:50:24.928671 2557 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:24.928756 2557 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9614d5590845a195b41d0c8c4cb0ff, State: Initialized, Role: FOLLOWER
I20260928 14:50:24.929302 2557 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260928 14:50:24.929551 2557 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:24.929649 2557 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:24.929767 2557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:24.931281 2557 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260928 14:50:24.931790 2557 leader_election.cc:302] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd9614d5590845a195b41d0c8c4cb0ff; no voters:
I20260928 14:50:24.932228 2557 leader_election.cc:288] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:24.932317 2567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:24.932544 2567 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [term 1 LEADER]: Becoming Leader. State: Replica: fd9614d5590845a195b41d0c8c4cb0ff, State: Running, Role: LEADER
I20260928 14:50:24.932978 2567 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } }
I20260928 14:50:24.933758 2557 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:24.934348 2567 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd9614d5590845a195b41d0c8c4cb0ff. Latest consensus state: current_term: 1 leader_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } } }
I20260928 14:50:24.934579 2567 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:24.934890 2568 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd9614d5590845a195b41d0c8c4cb0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33163 } } }
I20260928 14:50:24.935029 2568 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:24.935130 2572 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:24.936350 2560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:24.930402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55015 (local address 127.0.0.1:33163)
0928 14:50:24.930615 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:24.930622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:24.931084 (+ 462us) server_negotiation.cc:407] Connection header received
0928 14:50:24.931323 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:24.931327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:24.931385 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:24.931461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:24.931967 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:24.932846 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:24.933899 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:24.934225 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:24.934344 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:24.935571 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:24.935576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:24.935579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:24.935595 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:24.935640 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:24.935655 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:24.935731 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:24.936127 (+ 396us) server_negotiation.cc:299] Negotiation successful
0928 14:50:24.936195 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":877,"server-negotiator.queue_time_us":88}
I20260928 14:50:24.938761 2572 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:24.948164 2572 catalog_manager.cc:1347] Generated new cluster ID: a6c9342c32e7472598af396e7bf138ef
I20260928 14:50:24.948266 2572 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:24.992683 2572 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:24.994050 2572 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:25.004563 2572 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd9614d5590845a195b41d0c8c4cb0ff: Generated new TSK 0
I20260928 14:50:25.005460 2572 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:25.010468 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2487
2026-09-28T14:50:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1530 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:25 dist-test-slave-gkhx krb5kdc[2588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:25 dist-test-slave-gkhx krb5kdc[2588](info): set up 2 sockets
Sep 28 14:50:25 dist-test-slave-gkhx krb5kdc[2588](info): commencing operation
krb5kdc: starting...
W20260928 14:50:26.125360 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:26 dist-test-slave-gkhx krb5kdc[2588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:26.270907 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:26.389576 2601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:26.389914 2601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:26.389958 2601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:26.394327 2601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:26.394423 2601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:26.394443 2601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:26.394460 2601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:26.394476 2601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:26.400411 2601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38509
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:26.401657 2601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:26.402801 2601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:26.412369 2609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:26.412362 2607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:26.412437 2601 server_base.cc:1034] running on GCE node
W20260928 14:50:26.412362 2606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:26.413328 2601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:26.414719 2601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:26.416003 2601 hybrid_clock.cc:648] HybridClock initialized: now 1790607026415983 us; error 35 us; skew 500 ppm
Sep 28 14:50:26 dist-test-slave-gkhx krb5kdc[2588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:26.419332 2601 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:26.420421 2601 webserver.cc:460] Webserver started at http://127.0.0.1:46771/ using document root <none> and password file <none>
I20260928 14:50:26.421119 2601 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:26.421190 2601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:26.421428 2601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:26.423750 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "158abca885f741cf90128ebd160f6b74"
format_stamp: "Formatted at 2026-09-28 14:50:26 on dist-test-slave-gkhx"
server_key: "e36337cba993b12715635ca7279337d6"
server_key_iv: "7ea55757c9779d9b9085c246312fdee9"
server_key_version: "encryptionkey@0"
I20260928 14:50:26.424373 2601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "158abca885f741cf90128ebd160f6b74"
format_stamp: "Formatted at 2026-09-28 14:50:26 on dist-test-slave-gkhx"
server_key: "e36337cba993b12715635ca7279337d6"
server_key_iv: "7ea55757c9779d9b9085c246312fdee9"
server_key_version: "encryptionkey@0"
I20260928 14:50:26.428819 2601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 14:50:26.432237 2616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:26.434638 2601 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:26.434823 2601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "158abca885f741cf90128ebd160f6b74"
format_stamp: "Formatted at 2026-09-28 14:50:26 on dist-test-slave-gkhx"
server_key: "e36337cba993b12715635ca7279337d6"
server_key_iv: "7ea55757c9779d9b9085c246312fdee9"
server_key_version: "encryptionkey@0"
I20260928 14:50:26.434983 2601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:26.440941 2601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:26.444757 2601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:26.445016 2601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:26.454190 2601 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38509
I20260928 14:50:26.454200 2669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38509 every 8 connection(s)
I20260928 14:50:26.455410 2601 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:26.458776 2670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:26.459184 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2601
I20260928 14:50:26.459340 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:26.459654 14205 external_mini_cluster.cc:1428] Setting key c9491de183b99b0d3f49768d0db91dfc
I20260928 14:50:26.465927 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:26.461312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47653 (local address 127.0.0.1:38509)
0928 14:50:26.461848 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:26.461866 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:26.461906 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:50:26.462791 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:26.462814 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:26.463168 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:26.463582 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:26.464515 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:26.464521 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:26.464536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:26.464548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:26.464610 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:26.464630 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:26.464812 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:26.464891 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 14:50:26.465095 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":170,"threads_started":1}
I20260928 14:50:26.466717 2670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74: Bootstrap starting.
I20260928 14:50:26.469488 2670 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:26.469552 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:26.470336 2670 log.cc:824] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:26.473284 2670 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74: No bootstrap required, opened a new log
I20260928 14:50:26.474382 2673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:26.471983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43619 (local address 127.0.0.1:38509)
0928 14:50:26.472217 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:26.472222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:26.472549 (+ 327us) server_negotiation.cc:407] Connection header received
0928 14:50:26.472643 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:26.472645 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:26.472683 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:26.472763 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:26.473794 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:26.473797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:26.473799 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:26.473811 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:26.473840 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:26.473853 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:26.473940 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:26.474110 (+ 170us) server_negotiation.cc:299] Negotiation successful
0928 14:50:26.474202 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 14:50:26.477525 2670 raft_consensus.cc:348] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260928 14:50:26.477818 2670 raft_consensus.cc:374] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:26.477851 2670 raft_consensus.cc:729] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 158abca885f741cf90128ebd160f6b74, State: Initialized, Role: FOLLOWER
I20260928 14:50:26.478395 2670 consensus_queue.cc:260] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260928 14:50:26.478538 2670 raft_consensus.cc:388] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:26.478674 2670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:26.478813 2670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:26.479898 2670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260928 14:50:26.480307 2670 leader_election.cc:302] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 158abca885f741cf90128ebd160f6b74; no voters:
I20260928 14:50:26.480854 2670 leader_election.cc:288] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:26.480939 2680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:26.481168 2680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [term 1 LEADER]: Becoming Leader. State: Replica: 158abca885f741cf90128ebd160f6b74, State: Running, Role: LEADER
I20260928 14:50:26.481539 2680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } }
I20260928 14:50:26.482327 2670 sys_catalog.cc:564] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:26.483453 2681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 158abca885f741cf90128ebd160f6b74. Latest consensus state: current_term: 1 leader_uuid: "158abca885f741cf90128ebd160f6b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } } }
I20260928 14:50:26.483608 2681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:26.484021 2690 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:26.483986 2682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "158abca885f741cf90128ebd160f6b74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "158abca885f741cf90128ebd160f6b74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38509 } } }
I20260928 14:50:26.484159 2682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:26.487368 2690 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:26.493942 2690 catalog_manager.cc:1347] Generated new cluster ID: 92c8a072faa44a0eb06a719e5269a6af
I20260928 14:50:26.494155 2690 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:26.505620 2690 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:26.505970 2690 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:26.519850 2690 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 158abca885f741cf90128ebd160f6b74: Generated new TSK 0
I20260928 14:50:26.520936 2690 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:26.536237 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2601
2026-09-28T14:50:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1525 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:26 dist-test-slave-gkhx krb5kdc[2702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:26 dist-test-slave-gkhx krb5kdc[2702](info): set up 2 sockets
Sep 28 14:50:26 dist-test-slave-gkhx krb5kdc[2702](info): commencing operation
krb5kdc: starting...
W20260928 14:50:27.617128 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:27 dist-test-slave-gkhx krb5kdc[2702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:27.759434 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39127
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:27.880080 2715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:27.880419 2715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:27.880465 2715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:27.884716 2715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:27.884809 2715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:27.884832 2715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:27.884855 2715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:27.884874 2715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:27.890475 2715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39127
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:27.891670 2715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:27.892724 2715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:27.902840 2721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:27.902871 2720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:27.902909 2723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:27.903225 2715 server_base.cc:1034] running on GCE node
I20260928 14:50:27.903996 2715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:27.905198 2715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:27.906387 2715 hybrid_clock.cc:648] HybridClock initialized: now 1790607027906373 us; error 33 us; skew 500 ppm
Sep 28 14:50:27 dist-test-slave-gkhx krb5kdc[2702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:27.916558 2715 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:27.917753 2715 webserver.cc:460] Webserver started at http://127.0.0.1:42371/ using document root <none> and password file <none>
I20260928 14:50:27.918440 2715 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:27.918592 2715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:27.918884 2715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:27.921317 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "90c7570ff9d340fa8b62edd7ead3ef8d"
format_stamp: "Formatted at 2026-09-28 14:50:27 on dist-test-slave-gkhx"
server_key: "2e37ec8d13ed2fc53990c61dc16c2253"
server_key_iv: "eb657b6c0d5e4dc7e0c4ab50c62fe037"
server_key_version: "encryptionkey@0"
I20260928 14:50:27.922065 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "90c7570ff9d340fa8b62edd7ead3ef8d"
format_stamp: "Formatted at 2026-09-28 14:50:27 on dist-test-slave-gkhx"
server_key: "2e37ec8d13ed2fc53990c61dc16c2253"
server_key_iv: "eb657b6c0d5e4dc7e0c4ab50c62fe037"
server_key_version: "encryptionkey@0"
I20260928 14:50:27.926501 2715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 14:50:27.929615 2730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:27.931452 2715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:27.931674 2715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "90c7570ff9d340fa8b62edd7ead3ef8d"
format_stamp: "Formatted at 2026-09-28 14:50:27 on dist-test-slave-gkhx"
server_key: "2e37ec8d13ed2fc53990c61dc16c2253"
server_key_iv: "eb657b6c0d5e4dc7e0c4ab50c62fe037"
server_key_version: "encryptionkey@0"
I20260928 14:50:27.931833 2715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:27.947019 2715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:27.951097 2715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:27.951349 2715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:27.959919 2715 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36635
I20260928 14:50:27.960008 2783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36635 every 8 connection(s)
I20260928 14:50:27.961122 2715 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:27.964360 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:27.967948 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2715
I20260928 14:50:27.968127 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:27.968430 14205 external_mini_cluster.cc:1428] Setting key 041dc6a739c705ef13baec37eb460879
I20260928 14:50:27.971580 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d: Bootstrap starting.
I20260928 14:50:27.974361 2784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:27.975370 2784 log.cc:824] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:27.978176 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d: No bootstrap required, opened a new log
Sep 28 14:50:27 dist-test-slave-gkhx krb5kdc[2702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:50:27.982286 2784 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260928 14:50:27.982621 2784 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:27.982710 2784 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90c7570ff9d340fa8b62edd7ead3ef8d, State: Initialized, Role: FOLLOWER
I20260928 14:50:27.983291 2784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260928 14:50:27.983484 2784 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:27.983582 2784 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:27.983696 2784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:27.985095 2784 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260928 14:50:27.985513 2784 leader_election.cc:302] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90c7570ff9d340fa8b62edd7ead3ef8d; no voters:
I20260928 14:50:27.985878 2784 leader_election.cc:288] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:27.986012 2789 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:27.986248 2789 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [term 1 LEADER]: Becoming Leader. State: Replica: 90c7570ff9d340fa8b62edd7ead3ef8d, State: Running, Role: LEADER
I20260928 14:50:27.986495 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:27.970121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:45159 (local address 127.0.0.1:36635)
0928 14:50:27.970700 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:27.970724 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:27.970775 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:50:27.971571 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:27.971592 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:27.972081 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:27.972416 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:27.973442 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:27.974783 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:27.976042 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:27.976782 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:27.976933 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:27.980219 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:27.980226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:27.980243 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:27.980279 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:27.982374 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:27.983211 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:27.983218 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:27.983224 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:27.983391 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:27.983910 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:27.983916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:27.983921 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:27.984490 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:27.984738 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:27.985087 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 14:50:27.985352 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":203,"threads_started":1}
I20260928 14:50:27.986698 2789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } }
I20260928 14:50:27.987319 2784 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:27.988415 2790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260928 14:50:27.988559 2790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:27.989696 2791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90c7570ff9d340fa8b62edd7ead3ef8d. Latest consensus state: current_term: 1 leader_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90c7570ff9d340fa8b62edd7ead3ef8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36635 } } }
I20260928 14:50:27.989816 2791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:27.990168 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:27.990242 2798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:27.994393 2798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:28.002666 2787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:27.992840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:48135 (local address 127.0.0.1:36635)
0928 14:50:27.992977 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:27.992985 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:27.993006 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:50:27.993177 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:27.993180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:27.993243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:27.993351 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:27.994192 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:27.995013 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:27.996181 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:27.996443 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:27.996568 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:27.998146 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:27.998157 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:27.998160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:27.998189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:28.000239 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:28.001016 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:28.001020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:28.001022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:28.001143 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:28.001559 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:28.001563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:28.001565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:28.001830 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:28.001991 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:28.002325 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 14:50:28.002448 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 14:50:28.002909 2798 catalog_manager.cc:1347] Generated new cluster ID: 249c3d0e437c4baeb544b855d8372b48
I20260928 14:50:28.002969 2798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:28.023698 2798 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:28.025218 2798 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:28.037117 2798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90c7570ff9d340fa8b62edd7ead3ef8d: Generated new TSK 0
I20260928 14:50:28.038147 2798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:28.048918 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2715
W20260928 14:50:28.055505 2700 connection.cc:535] client connection to 127.0.0.1:36635 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36635): Connection reset by peer (error 104)
2026-09-28T14:50:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1513 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:28 dist-test-slave-gkhx krb5kdc[2815](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:28 dist-test-slave-gkhx krb5kdc[2815](info): set up 2 sockets
Sep 28 14:50:28 dist-test-slave-gkhx krb5kdc[2815](info): commencing operation
krb5kdc: starting...
W20260928 14:50:29.165719 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:29 dist-test-slave-gkhx krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:29.311159 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:29.430968 2828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:29.431332 2828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:29.431416 2828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:29.435678 2828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:29.435781 2828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:29.435820 2828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:29.435879 2828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:29.435916 2828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:29.441696 2828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:29.442970 2828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:29.444056 2828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:29.453903 2834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:29.453933 2836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:29.454145 2828 server_base.cc:1034] running on GCE node
W20260928 14:50:29.453989 2833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:29.455104 2828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:29.456939 2828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:29.458199 2828 hybrid_clock.cc:648] HybridClock initialized: now 1790607029458160 us; error 54 us; skew 500 ppm
Sep 28 14:50:29 dist-test-slave-gkhx krb5kdc[2815](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607029, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:29.461702 2828 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:29.462875 2828 webserver.cc:460] Webserver started at http://127.0.0.1:35181/ using document root <none> and password file <none>
I20260928 14:50:29.463498 2828 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:29.463569 2828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:29.463757 2828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:29.466034 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "d13441c63c5e4749aa86c7a78e7ab422"
format_stamp: "Formatted at 2026-09-28 14:50:29 on dist-test-slave-gkhx"
server_key: "8bca11d01a68c161969f439beac0fbe3"
server_key_iv: "bd1f057a678ff7772c4bc81bdd448942"
server_key_version: "encryptionkey@0"
I20260928 14:50:29.466758 2828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "d13441c63c5e4749aa86c7a78e7ab422"
format_stamp: "Formatted at 2026-09-28 14:50:29 on dist-test-slave-gkhx"
server_key: "8bca11d01a68c161969f439beac0fbe3"
server_key_iv: "bd1f057a678ff7772c4bc81bdd448942"
server_key_version: "encryptionkey@0"
I20260928 14:50:29.471685 2828 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260928 14:50:29.474829 2843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:29.476511 2828 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:29.476742 2828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "d13441c63c5e4749aa86c7a78e7ab422"
format_stamp: "Formatted at 2026-09-28 14:50:29 on dist-test-slave-gkhx"
server_key: "8bca11d01a68c161969f439beac0fbe3"
server_key_iv: "bd1f057a678ff7772c4bc81bdd448942"
server_key_version: "encryptionkey@0"
I20260928 14:50:29.476893 2828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:29.503142 2828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:29.507236 2828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:29.507480 2828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:29.515436 2828 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38423
I20260928 14:50:29.515448 2895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38423 every 8 connection(s)
I20260928 14:50:29.516518 2828 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:29.519618 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2828
I20260928 14:50:29.519657 2896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:29.519750 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:29.520063 14205 external_mini_cluster.cc:1428] Setting key a1e03bfa3042eb4bbcb569b1c0ead1c9
I20260928 14:50:29.527441 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422: Bootstrap starting.
I20260928 14:50:29.529855 2896 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:29.530009 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:29.521553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:50633 (local address 127.0.0.1:38423)
0928 14:50:29.522024 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:29.522041 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:29.522077 (+ 36us) server_negotiation.cc:407] Connection header received
0928 14:50:29.522785 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:29.522803 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:29.523223 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:29.523553 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:29.523929 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:29.524931 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:29.526222 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:29.526811 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:29.526938 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:29.528187 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:29.528192 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:29.528207 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:29.528623 (+ 416us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:29.528686 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:29.528710 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:29.528857 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:29.529039 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 14:50:29.529197 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":156,"threads_started":1}
I20260928 14:50:29.530877 2896 log.cc:824] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:29.533146 2896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422: No bootstrap required, opened a new log
I20260928 14:50:29.533931 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:29.538246 2896 raft_consensus.cc:348] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260928 14:50:29.538627 2896 raft_consensus.cc:374] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:29.538697 2896 raft_consensus.cc:729] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d13441c63c5e4749aa86c7a78e7ab422, State: Initialized, Role: FOLLOWER
I20260928 14:50:29.539225 2896 consensus_queue.cc:260] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260928 14:50:29.539377 2896 raft_consensus.cc:388] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:29.539466 2896 raft_consensus.cc:482] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:29.539584 2896 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:29.541049 2896 raft_consensus.cc:504] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260928 14:50:29.541373 2899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:29.536385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:55535 (local address 127.0.0.1:38423)
0928 14:50:29.536530 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:29.536535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:29.536740 (+ 205us) server_negotiation.cc:407] Connection header received
0928 14:50:29.536969 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:29.536976 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:29.537038 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:29.537135 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:29.537482 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:29.538072 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:29.539235 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:29.539611 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:29.539740 (+ 129us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:29.540635 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:29.540650 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:29.540652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:29.540664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:29.540691 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:29.540701 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:29.540773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:29.541155 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 14:50:29.541203 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 14:50:29.541477 2896 leader_election.cc:302] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d13441c63c5e4749aa86c7a78e7ab422; no voters:
I20260928 14:50:29.541914 2896 leader_election.cc:288] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:29.541985 2906 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:29.542217 2906 raft_consensus.cc:686] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [term 1 LEADER]: Becoming Leader. State: Replica: d13441c63c5e4749aa86c7a78e7ab422, State: Running, Role: LEADER
I20260928 14:50:29.542541 2906 consensus_queue.cc:237] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } }
I20260928 14:50:29.543395 2896 sys_catalog.cc:564] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:29.544440 2907 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d13441c63c5e4749aa86c7a78e7ab422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } } }
I20260928 14:50:29.544579 2907 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:29.544898 2908 sys_catalog.cc:455] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d13441c63c5e4749aa86c7a78e7ab422. Latest consensus state: current_term: 1 leader_uuid: "d13441c63c5e4749aa86c7a78e7ab422" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d13441c63c5e4749aa86c7a78e7ab422" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38423 } } }
I20260928 14:50:29.544996 2908 sys_catalog.cc:458] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:29.545086 2915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:29.548753 2915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:29.554968 2915 catalog_manager.cc:1347] Generated new cluster ID: 045a5f423c054bb9873cd5b04779f393
I20260928 14:50:29.555058 2915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:29.568058 2915 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:29.569401 2915 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:29.577579 2915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d13441c63c5e4749aa86c7a78e7ab422: Generated new TSK 0
I20260928 14:50:29.578486 2915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:29.590252 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2828
W20260928 14:50:29.595970 2813 connection.cc:535] client connection to 127.0.0.1:38423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38423): Connection reset by peer (error 104)
2026-09-28T14:50:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1540 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:29 dist-test-slave-gkhx krb5kdc[2927](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:29 dist-test-slave-gkhx krb5kdc[2927](info): set up 2 sockets
Sep 28 14:50:29 dist-test-slave-gkhx krb5kdc[2927](info): commencing operation
krb5kdc: starting...
W20260928 14:50:30.697033 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:30 dist-test-slave-gkhx krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:30.840001 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39887
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:30.965145 2940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:30.965482 2940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:30.965528 2940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:30.970005 2940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:30.970103 2940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:30.970124 2940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:30.970140 2940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:30.970156 2940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:30.976100 2940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:30.977401 2940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:30.978514 2940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:50:30.988785 2940 server_base.cc:1034] running on GCE node
W20260928 14:50:30.988713 2945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:30.988706 2946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:30.988713 2948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:30.990051 2940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:30.991815 2940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:30.993155 2940 hybrid_clock.cc:648] HybridClock initialized: now 1790607030993100 us; error 76 us; skew 500 ppm
Sep 28 14:50:30 dist-test-slave-gkhx krb5kdc[2927](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:30.997222 2940 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:30.998636 2940 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
I20260928 14:50:30.999310 2940 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:30.999413 2940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:30.999812 2940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:31.002440 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "e55fb03084754814ade94e9213570db4"
format_stamp: "Formatted at 2026-09-28 14:50:30 on dist-test-slave-gkhx"
server_key: "2057ab299e82e6e09c9c8b419f179ccf"
server_key_iv: "b97827216a7a2e90f919123fc82218dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:31.003242 2940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "e55fb03084754814ade94e9213570db4"
format_stamp: "Formatted at 2026-09-28 14:50:30 on dist-test-slave-gkhx"
server_key: "2057ab299e82e6e09c9c8b419f179ccf"
server_key_iv: "b97827216a7a2e90f919123fc82218dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:31.008030 2940 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 14:50:31.011178 2955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:31.012701 2940 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:50:31.012890 2940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "e55fb03084754814ade94e9213570db4"
format_stamp: "Formatted at 2026-09-28 14:50:30 on dist-test-slave-gkhx"
server_key: "2057ab299e82e6e09c9c8b419f179ccf"
server_key_iv: "b97827216a7a2e90f919123fc82218dd"
server_key_version: "encryptionkey@0"
I20260928 14:50:31.013109 2940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:31.018390 2940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:31.022074 2940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:31.022409 2940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:31.030740 2940 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39887
I20260928 14:50:31.030768 3009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39887 every 8 connection(s)
I20260928 14:50:31.031848 2940 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:31.035346 3010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:31.038190 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 2940
I20260928 14:50:31.038388 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:31.038820 14205 external_mini_cluster.cc:1428] Setting key 0a7d8103b4a8cccab6b6a16bb53db6e5
I20260928 14:50:31.043381 3010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4: Bootstrap starting.
I20260928 14:50:31.045859 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:31.040468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:48885 (local address 127.0.0.1:39887)
0928 14:50:31.040999 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:31.041023 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:31.041077 (+ 54us) server_negotiation.cc:407] Connection header received
0928 14:50:31.042063 (+ 986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:31.042092 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:31.042539 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:31.043033 (+ 494us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:31.044147 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:31.044154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:31.044172 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:31.044187 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:31.044262 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:31.044293 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:31.044451 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:31.044610 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 14:50:31.044804 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":141,"threads_started":1}
I20260928 14:50:31.046594 3010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:31.047626 3010 log.cc:824] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:31.049669 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:31.050616 3010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4: No bootstrap required, opened a new log
I20260928 14:50:31.054190 3013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:31.052237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:59107 (local address 127.0.0.1:39887)
0928 14:50:31.052404 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:31.052409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:31.052671 (+ 262us) server_negotiation.cc:407] Connection header received
0928 14:50:31.052833 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:31.052835 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:31.052871 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:31.052935 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:31.053725 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:31.053728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:31.053730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:31.053741 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:31.053769 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:31.053781 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:31.053860 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:31.053970 (+ 110us) server_negotiation.cc:299] Negotiation successful
0928 14:50:31.054008 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:50:31.055697 3010 raft_consensus.cc:348] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260928 14:50:31.056001 3010 raft_consensus.cc:374] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:31.056063 3010 raft_consensus.cc:729] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e55fb03084754814ade94e9213570db4, State: Initialized, Role: FOLLOWER
I20260928 14:50:31.056519 3010 consensus_queue.cc:260] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260928 14:50:31.056679 3010 raft_consensus.cc:388] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:31.056756 3010 raft_consensus.cc:482] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:31.056861 3010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:31.057972 3010 raft_consensus.cc:504] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260928 14:50:31.058393 3010 leader_election.cc:302] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e55fb03084754814ade94e9213570db4; no voters:
I20260928 14:50:31.058818 3010 leader_election.cc:288] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:31.059170 3020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:31.059414 3020 raft_consensus.cc:686] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [term 1 LEADER]: Becoming Leader. State: Replica: e55fb03084754814ade94e9213570db4, State: Running, Role: LEADER
I20260928 14:50:31.059805 3020 consensus_queue.cc:237] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } }
I20260928 14:50:31.060900 3010 sys_catalog.cc:564] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:31.061961 3021 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e55fb03084754814ade94e9213570db4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } } }
I20260928 14:50:31.062026 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e55fb03084754814ade94e9213570db4. Latest consensus state: current_term: 1 leader_uuid: "e55fb03084754814ade94e9213570db4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e55fb03084754814ade94e9213570db4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39887 } } }
I20260928 14:50:31.062125 3021 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:31.062141 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:31.062569 3029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:31.066135 3029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:31.073583 3029 catalog_manager.cc:1347] Generated new cluster ID: 56d70b06eb5f4a069dbad4e111910ac5
I20260928 14:50:31.073706 3029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:31.096515 3029 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:31.096946 3029 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:31.108891 3029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e55fb03084754814ade94e9213570db4: Generated new TSK 0
I20260928 14:50:31.109767 3029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:31.120098 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 2940
2026-09-28T14:50:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1529 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:31 dist-test-slave-gkhx krb5kdc[3041](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:31 dist-test-slave-gkhx krb5kdc[3041](info): set up 2 sockets
Sep 28 14:50:31 dist-test-slave-gkhx krb5kdc[3041](info): commencing operation
krb5kdc: starting...
W20260928 14:50:32.200946 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:32.347999 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:32.469972 3054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:32.470360 3054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:32.470459 3054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:32.474960 3054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:32.475067 3054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:32.475104 3054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:32.475143 3054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:32.475199 3054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:32.481148 3054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37857
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:32.482427 3054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:32.483584 3054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:32.493425 3059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:32.493468 3062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:32.493502 3060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:32.493899 3054 server_base.cc:1034] running on GCE node
I20260928 14:50:32.494896 3054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:32.496316 3054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:32.497512 3054 hybrid_clock.cc:648] HybridClock initialized: now 1790607032497497 us; error 44 us; skew 500 ppm
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3041](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607032, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:32.501130 3054 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:32.502377 3054 webserver.cc:460] Webserver started at http://127.0.0.1:34621/ using document root <none> and password file <none>
I20260928 14:50:32.503093 3054 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:32.503168 3054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:32.503448 3054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:32.505928 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "c874dcee368f43e9b90b22902e1a11b8"
format_stamp: "Formatted at 2026-09-28 14:50:32 on dist-test-slave-gkhx"
server_key: "2d604e2090da2b315b84062b89d1c75a"
server_key_iv: "bda6c3019d34d0c888666f82e7b15826"
server_key_version: "encryptionkey@0"
I20260928 14:50:32.506700 3054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "c874dcee368f43e9b90b22902e1a11b8"
format_stamp: "Formatted at 2026-09-28 14:50:32 on dist-test-slave-gkhx"
server_key: "2d604e2090da2b315b84062b89d1c75a"
server_key_iv: "bda6c3019d34d0c888666f82e7b15826"
server_key_version: "encryptionkey@0"
I20260928 14:50:32.511233 3054 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:32.514241 3069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:32.515849 3054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:32.516084 3054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c874dcee368f43e9b90b22902e1a11b8"
format_stamp: "Formatted at 2026-09-28 14:50:32 on dist-test-slave-gkhx"
server_key: "2d604e2090da2b315b84062b89d1c75a"
server_key_iv: "bda6c3019d34d0c888666f82e7b15826"
server_key_version: "encryptionkey@0"
I20260928 14:50:32.516235 3054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:32.534960 3054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:32.538661 3054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:32.539029 3054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:32.547654 3054 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37857
I20260928 14:50:32.547681 3123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37857 every 8 connection(s)
I20260928 14:50:32.548807 3054 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:32.552156 3124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:32.556192 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3054
I20260928 14:50:32.556398 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:32.556704 14205 external_mini_cluster.cc:1428] Setting key 074a640abaf0011b71ae2c01a3fbed70
I20260928 14:50:32.559473 3124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8: Bootstrap starting.
I20260928 14:50:32.562690 3124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:32.563709 3124 log.cc:824] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:32.566346 3124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8: No bootstrap required, opened a new log
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3041](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 14:50:32.571261 3124 raft_consensus.cc:348] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260928 14:50:32.571658 3124 raft_consensus.cc:374] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:32.571724 3124 raft_consensus.cc:729] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c874dcee368f43e9b90b22902e1a11b8, State: Initialized, Role: FOLLOWER
I20260928 14:50:32.572328 3124 consensus_queue.cc:260] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260928 14:50:32.572485 3124 raft_consensus.cc:388] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:32.572583 3124 raft_consensus.cc:482] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:32.572732 3124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:32.574173 3124 raft_consensus.cc:504] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260928 14:50:32.574718 3124 leader_election.cc:302] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c874dcee368f43e9b90b22902e1a11b8; no voters:
I20260928 14:50:32.575189 3124 leader_election.cc:288] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:32.575371 3129 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:32.575631 3129 raft_consensus.cc:686] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [term 1 LEADER]: Becoming Leader. State: Replica: c874dcee368f43e9b90b22902e1a11b8, State: Running, Role: LEADER
I20260928 14:50:32.576042 3129 consensus_queue.cc:237] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } }
I20260928 14:50:32.576411 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:32.558378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:39125 (local address 127.0.0.1:37857)
0928 14:50:32.558948 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:32.558969 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:32.559020 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:50:32.559972 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:32.559999 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:32.560513 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:32.560979 (+ 466us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:32.561812 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:32.563614 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:32.564788 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:32.565571 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:32.565754 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:32.568813 (+ 3059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:32.568820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:32.568842 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:32.568879 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:32.571958 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:32.572862 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:32.572869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:32.572877 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:32.573071 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:32.573483 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:32.573486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:32.573488 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:32.574175 (+ 687us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:32.574417 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:32.574923 (+ 506us) server_negotiation.cc:299] Negotiation successful
0928 14:50:32.575228 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":201,"threads_started":1}
I20260928 14:50:32.576766 3124 sys_catalog.cc:564] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:32.578017 3131 sys_catalog.cc:455] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c874dcee368f43e9b90b22902e1a11b8. Latest consensus state: current_term: 1 leader_uuid: "c874dcee368f43e9b90b22902e1a11b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260928 14:50:32.578156 3131 sys_catalog.cc:458] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:32.578645 3130 sys_catalog.cc:455] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c874dcee368f43e9b90b22902e1a11b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c874dcee368f43e9b90b22902e1a11b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37857 } } }
I20260928 14:50:32.578758 3130 sys_catalog.cc:458] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:32.579135 3138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:32.580605 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:32.584007 3138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:32.592049 3138 catalog_manager.cc:1347] Generated new cluster ID: becc1647a8c54c80a524edd9b77410b4
I20260928 14:50:32.592130 3138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:32.594034 3127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:32.583180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:56601 (local address 127.0.0.1:37857)
0928 14:50:32.583326 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:32.583332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:32.583629 (+ 297us) server_negotiation.cc:407] Connection header received
0928 14:50:32.583757 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:32.583761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:32.583819 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:32.583924 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:32.584863 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:32.585705 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:32.586999 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:32.587420 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:32.587556 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:32.589565 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:32.589569 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:32.589572 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:32.589608 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:32.591643 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:32.592381 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:32.592386 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:32.592388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:32.592503 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:32.592924 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:32.592928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:32.592930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:32.593192 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:32.593351 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:32.593723 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 14:50:32.593836 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:50:32.630327 3138 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:32.631904 3138 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:32.639745 3138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c874dcee368f43e9b90b22902e1a11b8: Generated new TSK 0
I20260928 14:50:32.640753 3138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:32.651993 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3054
W20260928 14:50:32.658463 3039 connection.cc:535] client connection to 127.0.0.1:37857 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37857): Connection reset by peer (error 104)
2026-09-28T14:50:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1532 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3156](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3156](info): set up 2 sockets
Sep 28 14:50:32 dist-test-slave-gkhx krb5kdc[3156](info): commencing operation
krb5kdc: starting...
W20260928 14:50:33.765533 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:33 dist-test-slave-gkhx krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607033, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:33.907207 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:34.030961 3169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:34.031596 3169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:34.031740 3169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:34.036432 3169 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:34.036554 3169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:34.036590 3169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:34.036634 3169 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:34.036696 3169 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:34.042948 3169 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:34.044255 3169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:34.045383 3169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:34.055105 3175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:34.055341 3174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:34.055177 3177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:34.055248 3169 server_base.cc:1034] running on GCE node
I20260928 14:50:34.056042 3169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:34.057443 3169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:34.058735 3169 hybrid_clock.cc:648] HybridClock initialized: now 1790607034058652 us; error 100 us; skew 500 ppm
Sep 28 14:50:34 dist-test-slave-gkhx krb5kdc[3156](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607034, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:34.070371 3169 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:34.071707 3169 webserver.cc:460] Webserver started at http://127.0.0.1:38079/ using document root <none> and password file <none>
I20260928 14:50:34.072350 3169 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:34.072425 3169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:34.072643 3169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:34.075192 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "5290d6ab4f284570be59b2ebcbf5e20d"
format_stamp: "Formatted at 2026-09-28 14:50:34 on dist-test-slave-gkhx"
server_key: "75929c5b8145ddc9056f0f44cbf23789"
server_key_iv: "31d41809b181773027139f06c93486de"
server_key_version: "encryptionkey@0"
I20260928 14:50:34.075798 3169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "5290d6ab4f284570be59b2ebcbf5e20d"
format_stamp: "Formatted at 2026-09-28 14:50:34 on dist-test-slave-gkhx"
server_key: "75929c5b8145ddc9056f0f44cbf23789"
server_key_iv: "31d41809b181773027139f06c93486de"
server_key_version: "encryptionkey@0"
I20260928 14:50:34.080850 3169 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 14:50:34.084066 3184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:34.086012 3169 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:50:34.086174 3169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "5290d6ab4f284570be59b2ebcbf5e20d"
format_stamp: "Formatted at 2026-09-28 14:50:34 on dist-test-slave-gkhx"
server_key: "75929c5b8145ddc9056f0f44cbf23789"
server_key_iv: "31d41809b181773027139f06c93486de"
server_key_version: "encryptionkey@0"
I20260928 14:50:34.086366 3169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:34.101992 3169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:34.105829 3169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:34.106065 3169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:34.114634 3169 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43571
I20260928 14:50:34.114699 3237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43571 every 8 connection(s)
I20260928 14:50:34.115768 3169 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:34.118997 3238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:34.125739 3238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d: Bootstrap starting.
I20260928 14:50:34.125846 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3169
I20260928 14:50:34.126084 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:34.126405 14205 external_mini_cluster.cc:1428] Setting key 5fb8b671ab6ff7e32f45256ee1d81da3
I20260928 14:50:34.129067 3238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:34.130096 3238 log.cc:824] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:34.132488 3238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d: No bootstrap required, opened a new log
I20260928 14:50:34.136760 3238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } }
I20260928 14:50:34.137064 3238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:34.137125 3238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5290d6ab4f284570be59b2ebcbf5e20d, State: Initialized, Role: FOLLOWER
I20260928 14:50:34.137584 3238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } }
I20260928 14:50:34.137718 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:34.128071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:45463 (local address 127.0.0.1:43571)
0928 14:50:34.128594 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:34.128615 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:34.128663 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:50:34.129710 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:34.129738 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:34.130275 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:34.130746 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:34.131261 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:34.132375 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:34.133554 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:34.134139 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:34.134368 (+ 229us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:34.135511 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:34.135517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:34.135533 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:34.136006 (+ 473us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:34.136073 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:34.136109 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:34.136274 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:34.136488 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 14:50:34.136652 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":145,"threads_started":1}
I20260928 14:50:34.137744 3238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:34.137902 3238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:34.138039 3238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:34.139158 3238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } }
I20260928 14:50:34.139523 3238 leader_election.cc:302] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5290d6ab4f284570be59b2ebcbf5e20d; no voters:
I20260928 14:50:34.139904 3238 leader_election.cc:288] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:34.140056 3243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:34.140342 3243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [term 1 LEADER]: Becoming Leader. State: Replica: 5290d6ab4f284570be59b2ebcbf5e20d, State: Running, Role: LEADER
I20260928 14:50:34.140765 3243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } }
I20260928 14:50:34.141214 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:34.141350 3238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:34.142428 3244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } } }
I20260928 14:50:34.142585 3244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:34.142833 3244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5290d6ab4f284570be59b2ebcbf5e20d. Latest consensus state: current_term: 1 leader_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5290d6ab4f284570be59b2ebcbf5e20d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43571 } } }
I20260928 14:50:34.142915 3244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:34.143304 3255 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:34.146175 3255 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:34.149672 3241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:34.144065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:48387 (local address 127.0.0.1:43571)
0928 14:50:34.144199 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:34.144204 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:34.144721 (+ 517us) server_negotiation.cc:407] Connection header received
0928 14:50:34.144894 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:34.144897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:34.144963 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:34.145049 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:34.145452 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:34.146258 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:34.147514 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:34.147879 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:34.147991 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:34.149113 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:34.149116 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:34.149119 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:34.149134 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:34.149163 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:34.149176 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:34.149290 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:34.149457 (+ 167us) server_negotiation.cc:299] Negotiation successful
0928 14:50:34.149499 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 14:50:34.152987 3255 catalog_manager.cc:1347] Generated new cluster ID: 6f46f89d46084067a1704f6cc8b8c766
I20260928 14:50:34.153108 3255 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:34.167630 3255 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:34.168751 3255 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:34.176329 3255 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5290d6ab4f284570be59b2ebcbf5e20d: Generated new TSK 0
I20260928 14:50:34.177196 3255 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:34.186931 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3169
W20260928 14:50:34.193352 3153 connection.cc:535] client connection to 127.0.0.1:43571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43571): Connection reset by peer (error 104)
2026-09-28T14:50:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1534 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:34 dist-test-slave-gkhx krb5kdc[3269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:34 dist-test-slave-gkhx krb5kdc[3269](info): set up 2 sockets
Sep 28 14:50:34 dist-test-slave-gkhx krb5kdc[3269](info): commencing operation
krb5kdc: starting...
W20260928 14:50:35.309772 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.101s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:35 dist-test-slave-gkhx krb5kdc[3269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 14:50:35.446683 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:35.565017 3282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:35.565338 3282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:35.565383 3282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:35.569756 3282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:35.569849 3282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:35.569870 3282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:35.569888 3282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:35.569904 3282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:35.575748 3282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38897
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:35.576995 3282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:35.578116 3282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:35.588057 3290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:35.588027 3288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:35.588078 3287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:35.588256 3282 server_base.cc:1034] running on GCE node
I20260928 14:50:35.589043 3282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:35.590313 3282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:35.591550 3282 hybrid_clock.cc:648] HybridClock initialized: now 1790607035591541 us; error 74 us; skew 500 ppm
Sep 28 14:50:35 dist-test-slave-gkhx krb5kdc[3269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607035, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:35.594820 3282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 14:50:35.595897 3282 webserver.cc:460] Webserver started at http://127.0.0.1:33877/ using document root <none> and password file <none>
I20260928 14:50:35.596485 3282 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:35.596549 3282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:35.596724 3282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:35.599026 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "1d371f25dc014b3ca80e661ed61f92ac"
format_stamp: "Formatted at 2026-09-28 14:50:35 on dist-test-slave-gkhx"
server_key: "0b350de146946f98c5327553f20eee8d"
server_key_iv: "8a9950060ac4cc7b42e503eac4829267"
server_key_version: "encryptionkey@0"
I20260928 14:50:35.599591 3282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "1d371f25dc014b3ca80e661ed61f92ac"
format_stamp: "Formatted at 2026-09-28 14:50:35 on dist-test-slave-gkhx"
server_key: "0b350de146946f98c5327553f20eee8d"
server_key_iv: "8a9950060ac4cc7b42e503eac4829267"
server_key_version: "encryptionkey@0"
I20260928 14:50:35.603956 3282 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:35.607118 3298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:35.608682 3282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 14:50:35.608809 3282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "1d371f25dc014b3ca80e661ed61f92ac"
format_stamp: "Formatted at 2026-09-28 14:50:35 on dist-test-slave-gkhx"
server_key: "0b350de146946f98c5327553f20eee8d"
server_key_iv: "8a9950060ac4cc7b42e503eac4829267"
server_key_version: "encryptionkey@0"
I20260928 14:50:35.608924 3282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:35.614073 3282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:35.617720 3282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:35.617892 3282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:35.625715 3282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38897
I20260928 14:50:35.625891 3351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38897 every 8 connection(s)
I20260928 14:50:35.626863 3282 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:35.630059 3352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:35.634214 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3282
I20260928 14:50:35.634349 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:35.634749 14205 external_mini_cluster.cc:1428] Setting key 211f27cb6cbe45b2ef185f79d824c4a7
I20260928 14:50:35.636984 3352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac: Bootstrap starting.
I20260928 14:50:35.639739 3352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:35.640661 3352 log.cc:824] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:35.641213 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:35.636530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:41737 (local address 127.0.0.1:38897)
0928 14:50:35.637068 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:35.637085 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:35.637122 (+ 37us) server_negotiation.cc:407] Connection header received
0928 14:50:35.637892 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:35.637910 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:35.638238 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:35.638636 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:35.639513 (+ 877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:35.639519 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:35.639532 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:35.639544 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:35.639607 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:35.639628 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:35.639727 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:35.639958 (+ 231us) server_negotiation.cc:299] Negotiation successful
0928 14:50:35.640185 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":197,"threads_started":1}
I20260928 14:50:35.643009 3352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac: No bootstrap required, opened a new log
I20260928 14:50:35.644714 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:35.647573 3352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260928 14:50:35.647977 3352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:35.648056 3352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d371f25dc014b3ca80e661ed61f92ac, State: Initialized, Role: FOLLOWER
I20260928 14:50:35.648677 3352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260928 14:50:35.648835 3352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:35.648977 3352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:35.649104 3352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:35.649261 3355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:35.647200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.33:54151 (local address 127.0.0.1:38897)
0928 14:50:35.647357 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:35.647362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:35.647590 (+ 228us) server_negotiation.cc:407] Connection header received
0928 14:50:35.647732 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:35.647734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:35.647770 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:35.647845 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:35.648719 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:35.648722 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:35.648725 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:35.648736 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:35.648765 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:35.648778 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:35.648898 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:35.649003 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 14:50:35.649065 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:50:35.650609 3352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260928 14:50:35.650992 3352 leader_election.cc:302] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d371f25dc014b3ca80e661ed61f92ac; no voters:
I20260928 14:50:35.651321 3352 leader_election.cc:288] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:35.651494 3362 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:35.651809 3362 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [term 1 LEADER]: Becoming Leader. State: Replica: 1d371f25dc014b3ca80e661ed61f92ac, State: Running, Role: LEADER
I20260928 14:50:35.652201 3362 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260928 14:50:35.652736 3352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:35.654209 3363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d371f25dc014b3ca80e661ed61f92ac. Latest consensus state: current_term: 1 leader_uuid: "1d371f25dc014b3ca80e661ed61f92ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260928 14:50:35.654171 3364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d371f25dc014b3ca80e661ed61f92ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d371f25dc014b3ca80e661ed61f92ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260928 14:50:35.654333 3363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:35.654336 3364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:35.654830 3374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:35.658004 3374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:35.663828 3374 catalog_manager.cc:1347] Generated new cluster ID: 25a84d90bc9c4c638daee0702f939b9b
I20260928 14:50:35.663976 3374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:35.675927 3374 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:35.676358 3374 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:35.690912 3374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d371f25dc014b3ca80e661ed61f92ac: Generated new TSK 0
I20260928 14:50:35.691929 3374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:35.706377 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3282
2026-09-28T14:50:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1517 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:35 dist-test-slave-gkhx krb5kdc[3383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:35 dist-test-slave-gkhx krb5kdc[3383](info): set up 2 sockets
Sep 28 14:50:35 dist-test-slave-gkhx krb5kdc[3383](info): commencing operation
krb5kdc: starting...
W20260928 14:50:36.813045 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:36 dist-test-slave-gkhx krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:36.951152 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34751
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:42361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:37.074862 3396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:37.075292 3396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:37.075398 3396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:37.080192 3396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:37.080329 3396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:37.080375 3396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:37.080420 3396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:37.080454 3396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:37.086578 3396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:42361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34751
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:37.087850 3396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:37.088971 3396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:37.098619 3404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.098717 3402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:37.098619 3401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:37.099126 3396 server_base.cc:1034] running on GCE node
I20260928 14:50:37.099833 3396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:37.101282 3396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:37.102766 3396 hybrid_clock.cc:648] HybridClock initialized: now 1790607037102720 us; error 77 us; skew 500 ppm
Sep 28 14:50:37 dist-test-slave-gkhx krb5kdc[3383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607037, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:37.107268 3396 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:37.108490 3396 webserver.cc:460] Webserver started at http://127.13.223.126:37817/ using document root <none> and password file <none>
I20260928 14:50:37.109161 3396 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:37.109232 3396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:37.109474 3396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:37.112066 3396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "c2616311dfd84e0fa4e605e01b6cfc4b"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-gkhx"
server_key: "9ccd72e0ba31fe2fdc44193cda85c8bd"
server_key_iv: "02fc5edfba4e406c57f6e0837e611881"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.112792 3396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "c2616311dfd84e0fa4e605e01b6cfc4b"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-gkhx"
server_key: "9ccd72e0ba31fe2fdc44193cda85c8bd"
server_key_iv: "02fc5edfba4e406c57f6e0837e611881"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.117506 3396 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:37.121129 3411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:37.122941 3396 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 14:50:37.123129 3396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c2616311dfd84e0fa4e605e01b6cfc4b"
format_stamp: "Formatted at 2026-09-28 14:50:37 on dist-test-slave-gkhx"
server_key: "9ccd72e0ba31fe2fdc44193cda85c8bd"
server_key_iv: "02fc5edfba4e406c57f6e0837e611881"
server_key_version: "encryptionkey@0"
I20260928 14:50:37.123287 3396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:37.148059 3396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:37.152194 3396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:37.152433 3396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:37.160921 3396 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34751
I20260928 14:50:37.161104 3464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34751 every 8 connection(s)
I20260928 14:50:37.162030 3396 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:37.165376 3465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:37.169160 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3396
I20260928 14:50:37.169314 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:37.169641 14205 external_mini_cluster.cc:1428] Setting key b6e758ca901bd405f66e3316f0afe297
I20260928 14:50:37.172114 3465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: Bootstrap starting.
I20260928 14:50:37.174898 3465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:37.175725 3465 log.cc:824] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:37.178232 3465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: No bootstrap required, opened a new log
Sep 28 14:50:37 dist-test-slave-gkhx krb5kdc[3383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607036, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:37.183310 3465 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } }
I20260928 14:50:37.183672 3465 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:37.183738 3465 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2616311dfd84e0fa4e605e01b6cfc4b, State: Initialized, Role: FOLLOWER
I20260928 14:50:37.184309 3465 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } }
I20260928 14:50:37.184509 3465 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:37.184584 3465 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:37.184688 3465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:37.186371 3465 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } }
I20260928 14:50:37.186832 3465 leader_election.cc:302] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2616311dfd84e0fa4e605e01b6cfc4b; no voters:
I20260928 14:50:37.187211 3465 leader_election.cc:288] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:37.187350 3471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:37.187352 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.171432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49317 (local address 127.13.223.126:34751)
0928 14:50:37.171971 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.171989 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.172029 (+ 40us) server_negotiation.cc:407] Connection header received
0928 14:50:37.172824 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.172846 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.173270 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.173611 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.174725 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.175959 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.177136 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.177667 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.177810 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.181005 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.181013 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.181035 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.181078 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.183303 (+ 2225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.184061 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.184068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.184074 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.184226 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.184670 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.184675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.184677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.185190 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.185377 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.185782 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.186064 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":176,"threads_started":1}
I20260928 14:50:37.187791 3471 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [term 1 LEADER]: Becoming Leader. State: Replica: c2616311dfd84e0fa4e605e01b6cfc4b, State: Running, Role: LEADER
I20260928 14:50:37.188413 3471 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } }
I20260928 14:50:37.188725 3465 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:37.190294 3472 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } } }
I20260928 14:50:37.190443 3472 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:37.190904 3473 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2616311dfd84e0fa4e605e01b6cfc4b. Latest consensus state: current_term: 1 leader_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2616311dfd84e0fa4e605e01b6cfc4b" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34751 } } }
I20260928 14:50:37.191097 3473 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:37.192157 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 14:50:37.194273 3486 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:37.194396 3486 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:37.194494 3482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:37.195557 3482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:37.202219 3482 catalog_manager.cc:1347] Generated new cluster ID: 2f66303a2cbc47909e2ff0a310e69da8
I20260928 14:50:37.202306 3482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:37.208232 3469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:37.197430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40541 (local address 127.13.223.126:34751)
0928 14:50:37.197609 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:37.197618 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:37.197639 (+ 21us) server_negotiation.cc:407] Connection header received
0928 14:50:37.197689 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:37.197692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:37.197746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:37.197868 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:37.198929 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.200056 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.201327 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:37.201783 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:37.201908 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:37.204170 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:37.204174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:37.204177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:37.204209 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:37.206015 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.206730 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.206733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.206735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.206826 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:37.207213 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:37.207216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:37.207234 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:37.207447 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:37.207571 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:37.207973 (+ 402us) server_negotiation.cc:299] Negotiation successful
0928 14:50:37.208071 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260928 14:50:37.227550 3482 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:37.229031 3482 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:37.238950 3482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2616311dfd84e0fa4e605e01b6cfc4b: Generated new TSK 0
I20260928 14:50:37.239858 3482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:37.250962 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3396
W20260928 14:50:37.256551 3381 connection.cc:535] client connection to 127.13.223.126:34751 recv error: Network error: failed to read from TLS socket (remote: 127.13.223.126:34751): Connection reset by peer (error 104)
2026-09-28T14:50:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1544 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:37 dist-test-slave-gkhx krb5kdc[3497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:37 dist-test-slave-gkhx krb5kdc[3497](info): set up 2 sockets
Sep 28 14:50:37 dist-test-slave-gkhx krb5kdc[3497](info): commencing operation
krb5kdc: starting...
W20260928 14:50:38.328835 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:38 dist-test-slave-gkhx krb5kdc[3497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:38.471539 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:38905
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:38905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:38.592871 3510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:38.593223 3510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:38.593269 3510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:38.597666 3510 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:38.597769 3510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:38.597793 3510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:38.597815 3510 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:38.597833 3510 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:38.603994 3510 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:38905
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:38905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3510
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:38.605263 3510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:38.606424 3510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:38.616093 3518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:38.616312 3510 server_base.cc:1034] running on GCE node
W20260928 14:50:38.616107 3515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:38.616096 3516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:38.617154 3510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:38.618398 3510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:38.619643 3510 hybrid_clock.cc:648] HybridClock initialized: now 1790607038619605 us; error 53 us; skew 500 ppm
Sep 28 14:50:38 dist-test-slave-gkhx krb5kdc[3497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607038, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:38.622759 3510 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:38.623811 3510 webserver.cc:460] Webserver started at http://127.13.223.126:36267/ using document root <none> and password file <none>
I20260928 14:50:38.624419 3510 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:38.624516 3510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:38.624768 3510 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:38.627046 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "3124c9db984d408d986aa4426cea79ec"
format_stamp: "Formatted at 2026-09-28 14:50:38 on dist-test-slave-gkhx"
server_key: "f311221df1e67b8dafda450eec03f04d"
server_key_iv: "8eed1bdcc21464a4df4031cf5bcc649d"
server_key_version: "encryptionkey@0"
I20260928 14:50:38.627674 3510 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "3124c9db984d408d986aa4426cea79ec"
format_stamp: "Formatted at 2026-09-28 14:50:38 on dist-test-slave-gkhx"
server_key: "f311221df1e67b8dafda450eec03f04d"
server_key_iv: "8eed1bdcc21464a4df4031cf5bcc649d"
server_key_version: "encryptionkey@0"
I20260928 14:50:38.631878 3510 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260928 14:50:38.634703 3525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:38.636152 3510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:38.636317 3510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "3124c9db984d408d986aa4426cea79ec"
format_stamp: "Formatted at 2026-09-28 14:50:38 on dist-test-slave-gkhx"
server_key: "f311221df1e67b8dafda450eec03f04d"
server_key_iv: "8eed1bdcc21464a4df4031cf5bcc649d"
server_key_version: "encryptionkey@0"
I20260928 14:50:38.636452 3510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:38.667331 3510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:38.671247 3510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:38.671519 3510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:38.679736 3510 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:38905
I20260928 14:50:38.679751 3578 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:38905 every 8 connection(s)
I20260928 14:50:38.680866 3510 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:38.683903 3579 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:38.690091 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3510
I20260928 14:50:38.690125 3579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: Bootstrap starting.
I20260928 14:50:38.690236 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:38.690606 14205 external_mini_cluster.cc:1428] Setting key d93b0837dbcc51a785f06f24c629da67
I20260928 14:50:38.693018 3579 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:38.693812 3579 log.cc:824] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:38.696195 3579 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: No bootstrap required, opened a new log
I20260928 14:50:38.701074 3579 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } }
I20260928 14:50:38.701295 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:38.692120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36833 (local address 127.13.223.126:38905)
0928 14:50:38.692620 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:38.692637 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:38.692686 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:50:38.693495 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:38.693515 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:38.693887 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:38.694294 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:38.694815 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:38.695818 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:38.697124 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:38.697658 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:38.697801 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:38.698841 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:38.698848 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:38.698866 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:38.699482 (+ 616us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:38.699590 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:38.699634 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:38.699783 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:38.700078 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 14:50:38.700241 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":166,"threads_started":1}
I20260928 14:50:38.701390 3579 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:38.701437 3579 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3124c9db984d408d986aa4426cea79ec, State: Initialized, Role: FOLLOWER
I20260928 14:50:38.701941 3579 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } }
I20260928 14:50:38.702128 3579 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:38.702185 3579 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:38.702342 3579 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:38.703461 3579 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } }
I20260928 14:50:38.703862 3579 leader_election.cc:302] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3124c9db984d408d986aa4426cea79ec; no voters:
I20260928 14:50:38.704272 3579 leader_election.cc:288] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:38.704452 3584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:38.704933 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:38.704967 3584 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [term 1 LEADER]: Becoming Leader. State: Replica: 3124c9db984d408d986aa4426cea79ec, State: Running, Role: LEADER
I20260928 14:50:38.705331 3584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } }
I20260928 14:50:38.705593 3579 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:38.707327 3586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3124c9db984d408d986aa4426cea79ec. Latest consensus state: current_term: 1 leader_uuid: "3124c9db984d408d986aa4426cea79ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } } }
I20260928 14:50:38.707501 3586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:38.707901 3585 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3124c9db984d408d986aa4426cea79ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3124c9db984d408d986aa4426cea79ec" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38905 } } }
I20260928 14:50:38.708216 3585 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec [sys.catalog]: This master's current role is: LEADER
W20260928 14:50:38.711956 3604 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:38.712121 3604 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:38.712225 3603 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:38.713263 3603 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:38.713675 3582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:38.708265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60067 (local address 127.13.223.126:38905)
0928 14:50:38.708418 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:38.708426 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:38.708898 (+ 472us) server_negotiation.cc:407] Connection header received
0928 14:50:38.709136 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:38.709139 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:38.709186 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:38.709280 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:38.709626 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:38.710248 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:38.711453 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:38.711806 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:38.711923 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:38.713068 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:38.713073 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:38.713076 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:38.713092 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:38.713129 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:38.713143 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:38.713241 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:38.713482 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 14:50:38.713528 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:50:38.719969 3603 catalog_manager.cc:1347] Generated new cluster ID: 6ff9a1ffb1a64751a6e7be1a102a3791
I20260928 14:50:38.720072 3603 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:38.729447 3603 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:38.730711 3603 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:38.742905 3603 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3124c9db984d408d986aa4426cea79ec: Generated new TSK 0
I20260928 14:50:38.743855 3603 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:38.750700 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3510
W20260928 14:50:38.757380 3495 connection.cc:535] client connection to 127.13.223.126:38905 recv error: Network error: failed to read from TLS socket (remote: 127.13.223.126:38905): Connection reset by peer (error 104)
2026-09-28T14:50:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1500 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:38 dist-test-slave-gkhx krb5kdc[3611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:38 dist-test-slave-gkhx krb5kdc[3611](info): set up 2 sockets
Sep 28 14:50:38 dist-test-slave-gkhx krb5kdc[3611](info): commencing operation
krb5kdc: starting...
W20260928 14:50:39.832851 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:39 dist-test-slave-gkhx krb5kdc[3611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:39.976584 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:45065
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:46599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:45065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:40.095691 3624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:40.096063 3624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:40.096148 3624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:40.100497 3624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:40.100641 3624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:40.100678 3624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:40.100720 3624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:40.100754 3624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:40.106719 3624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:46599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:45065
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:45065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:40.107976 3624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:40.109155 3624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:40.118683 3630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:40.118683 3632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:40.118686 3629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:40.119118 3624 server_base.cc:1034] running on GCE node
I20260928 14:50:40.119841 3624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:40.121295 3624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:40.122625 3624 hybrid_clock.cc:648] HybridClock initialized: now 1790607040122516 us; error 54 us; skew 500 ppm
Sep 28 14:50:40 dist-test-slave-gkhx krb5kdc[3611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607040, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:40.134403 3624 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:40.135576 3624 webserver.cc:460] Webserver started at http://127.13.223.126:42079/ using document root <none> and password file <none>
I20260928 14:50:40.136286 3624 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:40.136391 3624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:40.136653 3624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:40.138988 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "c37545717462464699f084b61c91c52d"
format_stamp: "Formatted at 2026-09-28 14:50:40 on dist-test-slave-gkhx"
server_key: "19c81d358ddef732067e46591000a661"
server_key_iv: "c9b05498b89076fbae10cd2a20b2d2a1"
server_key_version: "encryptionkey@0"
I20260928 14:50:40.139640 3624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "c37545717462464699f084b61c91c52d"
format_stamp: "Formatted at 2026-09-28 14:50:40 on dist-test-slave-gkhx"
server_key: "19c81d358ddef732067e46591000a661"
server_key_iv: "c9b05498b89076fbae10cd2a20b2d2a1"
server_key_version: "encryptionkey@0"
I20260928 14:50:40.144052 3624 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:50:40.147217 3639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:40.148857 3624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260928 14:50:40.149112 3624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "c37545717462464699f084b61c91c52d"
format_stamp: "Formatted at 2026-09-28 14:50:40 on dist-test-slave-gkhx"
server_key: "19c81d358ddef732067e46591000a661"
server_key_iv: "c9b05498b89076fbae10cd2a20b2d2a1"
server_key_version: "encryptionkey@0"
I20260928 14:50:40.149264 3624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:40.154577 3624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:40.158017 3624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:40.158252 3624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:40.165936 3624 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:45065
I20260928 14:50:40.165962 3691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:45065 every 8 connection(s)
I20260928 14:50:40.167004 3624 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:40.169983 3692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:40.174767 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3624
I20260928 14:50:40.175086 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:40.175385 14205 external_mini_cluster.cc:1428] Setting key 33e2371fa7f4dd182c546c733a2a8c4b
I20260928 14:50:40.177011 3692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d: Bootstrap starting.
I20260928 14:50:40.179705 3692 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:40.180567 3692 log.cc:824] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:40.181707 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:40.177105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43263 (local address 127.13.223.126:45065)
0928 14:50:40.177660 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:40.177678 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:40.177729 (+ 51us) server_negotiation.cc:407] Connection header received
0928 14:50:40.178454 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:40.178474 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:40.178909 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:40.179374 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:40.180187 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:40.180191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:40.180204 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:40.180214 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:40.180266 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:40.180284 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:40.180405 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:40.180518 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 14:50:40.180735 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":148,"threads_started":1}
I20260928 14:50:40.183028 3692 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d: No bootstrap required, opened a new log
I20260928 14:50:40.185055 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:40.187444 3692 raft_consensus.cc:348] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } }
I20260928 14:50:40.187800 3692 raft_consensus.cc:374] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:40.187883 3692 raft_consensus.cc:729] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c37545717462464699f084b61c91c52d, State: Initialized, Role: FOLLOWER
I20260928 14:50:40.188436 3692 consensus_queue.cc:260] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } }
I20260928 14:50:40.188580 3692 raft_consensus.cc:388] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:40.188654 3692 raft_consensus.cc:482] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:40.188771 3692 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:40.189559 3695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:40.187305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40375 (local address 127.13.223.126:45065)
0928 14:50:40.187455 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:40.187460 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:40.187668 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:50:40.187901 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:40.187904 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:40.188040 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:40.188120 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:40.189104 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:40.189110 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:40.189116 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:40.189133 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:40.189167 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:40.189182 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:40.189287 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:40.189387 (+ 100us) server_negotiation.cc:299] Negotiation successful
0928 14:50:40.189431 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 14:50:40.190168 3692 raft_consensus.cc:504] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } }
I20260928 14:50:40.190650 3692 leader_election.cc:302] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c37545717462464699f084b61c91c52d; no voters:
I20260928 14:50:40.190999 3692 leader_election.cc:288] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:40.191183 3702 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:40.191512 3702 raft_consensus.cc:686] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [term 1 LEADER]: Becoming Leader. State: Replica: c37545717462464699f084b61c91c52d, State: Running, Role: LEADER
I20260928 14:50:40.192000 3702 consensus_queue.cc:237] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } }
I20260928 14:50:40.192320 3692 sys_catalog.cc:564] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:40.193817 3703 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c37545717462464699f084b61c91c52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } } }
I20260928 14:50:40.193981 3703 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:40.194414 3714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:40.194415 3704 sys_catalog.cc:455] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c37545717462464699f084b61c91c52d. Latest consensus state: current_term: 1 leader_uuid: "c37545717462464699f084b61c91c52d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c37545717462464699f084b61c91c52d" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 45065 } } }
I20260928 14:50:40.196935 3704 sys_catalog.cc:458] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:40.200104 3714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:40.210819 3714 catalog_manager.cc:1347] Generated new cluster ID: 0836b6e3ad964932ab9c61d0710f1317
I20260928 14:50:40.210922 3714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:40.232110 3714 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:40.232513 3714 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:40.244513 3714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c37545717462464699f084b61c91c52d: Generated new TSK 0
I20260928 14:50:40.245505 3714 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:40.257019 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3624
2026-09-28T14:50:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1505 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:40 dist-test-slave-gkhx krb5kdc[3723](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:40 dist-test-slave-gkhx krb5kdc[3723](info): set up 2 sockets
Sep 28 14:50:40 dist-test-slave-gkhx krb5kdc[3723](info): commencing operation
krb5kdc: starting...
W20260928 14:50:41.309696 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:41.455474 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34539
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:34539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:41.575302 3736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:41.575688 3736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:41.575764 3736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:41.580231 3736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:41.580343 3736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:41.580384 3736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:41.580442 3736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:41.580480 3736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:41.586675 3736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:34539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:34539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:41.588255 3736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:41.589715 3736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:41.599687 3742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:41.599694 3741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:41.600193 3736 server_base.cc:1034] running on GCE node
W20260928 14:50:41.599687 3744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:41.600879 3736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:41.602134 3736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:41.603358 3736 hybrid_clock.cc:648] HybridClock initialized: now 1790607041603301 us; error 73 us; skew 500 ppm
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3723](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607041, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:41.606935 3736 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:41.608143 3736 webserver.cc:460] Webserver started at http://127.13.223.126:33441/ using document root <none> and password file <none>
I20260928 14:50:41.608788 3736 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:41.608891 3736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:41.609184 3736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:41.611569 3736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "b17e858291cf4a6f921bc2b696491704"
format_stamp: "Formatted at 2026-09-28 14:50:41 on dist-test-slave-gkhx"
server_key: "ef23b2f175c484d79e57003b732ea785"
server_key_iv: "741058aeeb9ed5ace8241ef9fe4e3067"
server_key_version: "encryptionkey@0"
I20260928 14:50:41.612241 3736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "b17e858291cf4a6f921bc2b696491704"
format_stamp: "Formatted at 2026-09-28 14:50:41 on dist-test-slave-gkhx"
server_key: "ef23b2f175c484d79e57003b732ea785"
server_key_iv: "741058aeeb9ed5ace8241ef9fe4e3067"
server_key_version: "encryptionkey@0"
I20260928 14:50:41.616683 3736 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 14:50:41.619746 3751 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:41.621693 3736 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 14:50:41.621914 3736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "b17e858291cf4a6f921bc2b696491704"
format_stamp: "Formatted at 2026-09-28 14:50:41 on dist-test-slave-gkhx"
server_key: "ef23b2f175c484d79e57003b732ea785"
server_key_iv: "741058aeeb9ed5ace8241ef9fe4e3067"
server_key_version: "encryptionkey@0"
I20260928 14:50:41.622073 3736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:41.651249 3736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:41.655123 3736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:41.655475 3736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:41.664386 3736 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:34539
I20260928 14:50:41.664443 3803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:34539 every 8 connection(s)
I20260928 14:50:41.665721 3736 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:41.669147 3804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:41.674183 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3736
I20260928 14:50:41.674352 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:41.674681 14205 external_mini_cluster.cc:1428] Setting key c50998db5feeaefdb47d2a1159048daf
I20260928 14:50:41.675987 3804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704: Bootstrap starting.
I20260928 14:50:41.678838 3804 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:41.679941 3804 log.cc:824] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:41.682507 3804 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704: No bootstrap required, opened a new log
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3723](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:41.687469 3804 raft_consensus.cc:348] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } }
I20260928 14:50:41.687825 3804 raft_consensus.cc:374] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:41.687912 3804 raft_consensus.cc:729] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b17e858291cf4a6f921bc2b696491704, State: Initialized, Role: FOLLOWER
I20260928 14:50:41.688476 3804 consensus_queue.cc:260] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } }
I20260928 14:50:41.688649 3804 raft_consensus.cc:388] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:41.688725 3804 raft_consensus.cc:482] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:41.688858 3804 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:41.689880 3804 raft_consensus.cc:504] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } }
I20260928 14:50:41.690272 3804 leader_election.cc:302] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b17e858291cf4a6f921bc2b696491704; no voters:
I20260928 14:50:41.690704 3804 leader_election.cc:288] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:41.690858 3810 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:41.691113 3810 raft_consensus.cc:686] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [term 1 LEADER]: Becoming Leader. State: Replica: b17e858291cf4a6f921bc2b696491704, State: Running, Role: LEADER
I20260928 14:50:41.691464 3810 consensus_queue.cc:237] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } }
I20260928 14:50:41.692164 3804 sys_catalog.cc:564] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:41.692420 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:41.676523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60073 (local address 127.13.223.126:34539)
0928 14:50:41.677091 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:41.677109 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:41.677152 (+ 43us) server_negotiation.cc:407] Connection header received
0928 14:50:41.677935 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:41.677956 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:41.678367 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:41.678780 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:41.679828 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:41.681080 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:41.682324 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:41.682959 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:41.683115 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:41.686235 (+ 3120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:41.686242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:41.686258 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:41.686295 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:41.688485 (+ 2190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:41.689097 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:41.689103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:41.689109 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:41.689238 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:41.689570 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:41.689574 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:41.689577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:41.690072 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:41.690293 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:41.690676 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 14:50:41.691168 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":190,"threads_started":1}
I20260928 14:50:41.693567 3811 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b17e858291cf4a6f921bc2b696491704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } } }
I20260928 14:50:41.693722 3811 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:41.693552 3812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b17e858291cf4a6f921bc2b696491704. Latest consensus state: current_term: 1 leader_uuid: "b17e858291cf4a6f921bc2b696491704" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b17e858291cf4a6f921bc2b696491704" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 34539 } } }
I20260928 14:50:41.693812 3812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:41.694144 3820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:41.696059 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:41.698756 3820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:41.704977 3820 catalog_manager.cc:1347] Generated new cluster ID: 03b27d7890d140c48d9845aa5ea938fb
I20260928 14:50:41.705075 3820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:41.710144 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:41.698477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49749 (local address 127.13.223.126:34539)
0928 14:50:41.698640 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:41.698645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:41.701261 (+ 2616us) server_negotiation.cc:407] Connection header received
0928 14:50:41.701389 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:41.701393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:41.701451 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:41.701560 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:41.702534 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:41.703187 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:41.704473 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:41.704843 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:41.704950 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:41.706713 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:41.706718 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:41.706726 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:41.706754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:41.708323 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:41.708804 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:41.708807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:41.708809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:41.708901 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:41.709206 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:41.709210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:41.709211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:41.709427 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:41.709577 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:41.709826 (+ 249us) server_negotiation.cc:299] Negotiation successful
0928 14:50:41.709953 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":55}
I20260928 14:50:41.726334 3820 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:41.727723 3820 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:41.738425 3820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b17e858291cf4a6f921bc2b696491704: Generated new TSK 0
I20260928 14:50:41.739456 3820 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:41.752509 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3736
2026-09-28T14:50:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1496 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3836](info): set up 2 sockets
Sep 28 14:50:41 dist-test-slave-gkhx krb5kdc[3836](info): commencing operation
krb5kdc: starting...
W20260928 14:50:42.844995 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:42 dist-test-slave-gkhx krb5kdc[3836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607042, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:42.985962 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:38239
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:37831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:38239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:43.110044 3849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:43.110435 3849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:43.110520 3849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:43.115003 3849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:43.115136 3849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:43.115172 3849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:43.115208 3849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:43.115242 3849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:43.121042 3849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:37831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:38239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:38239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:43.122397 3849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:43.123625 3849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:43.133262 3857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:43.133265 3855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:43.133240 3849 server_base.cc:1034] running on GCE node
W20260928 14:50:43.133222 3854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:43.134253 3849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:43.135560 3849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:43.136767 3849 hybrid_clock.cc:648] HybridClock initialized: now 1790607043136736 us; error 58 us; skew 500 ppm
Sep 28 14:50:43 dist-test-slave-gkhx krb5kdc[3836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607043, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:43.140096 3849 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:43.141170 3849 webserver.cc:460] Webserver started at http://127.13.223.126:34591/ using document root <none> and password file <none>
I20260928 14:50:43.141822 3849 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:43.141921 3849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:43.142172 3849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:43.144451 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "8a89c44cff4e45a287ab56ab0d3213a1"
format_stamp: "Formatted at 2026-09-28 14:50:43 on dist-test-slave-gkhx"
server_key: "59afd769f94ccb265e4911a69bf85e90"
server_key_iv: "abf7e16970457a0950f52f42c79b34aa"
server_key_version: "encryptionkey@0"
I20260928 14:50:43.145115 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "8a89c44cff4e45a287ab56ab0d3213a1"
format_stamp: "Formatted at 2026-09-28 14:50:43 on dist-test-slave-gkhx"
server_key: "59afd769f94ccb265e4911a69bf85e90"
server_key_iv: "abf7e16970457a0950f52f42c79b34aa"
server_key_version: "encryptionkey@0"
I20260928 14:50:43.149449 3849 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:43.152742 3864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:43.154541 3849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 14:50:43.154769 3849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "8a89c44cff4e45a287ab56ab0d3213a1"
format_stamp: "Formatted at 2026-09-28 14:50:43 on dist-test-slave-gkhx"
server_key: "59afd769f94ccb265e4911a69bf85e90"
server_key_iv: "abf7e16970457a0950f52f42c79b34aa"
server_key_version: "encryptionkey@0"
I20260928 14:50:43.154925 3849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:43.175010 3849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:43.178445 3849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:43.178761 3849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:43.187593 3849 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:38239
I20260928 14:50:43.187651 3919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:38239 every 8 connection(s)
I20260928 14:50:43.188676 3849 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:43.191956 3920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:43.194288 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3849
I20260928 14:50:43.194471 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:43.194831 14205 external_mini_cluster.cc:1428] Setting key 7385fd43d366e10c74633b8cb1d274ba
I20260928 14:50:43.200469 3920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1: Bootstrap starting.
I20260928 14:50:43.203655 3920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:43.204726 3920 log.cc:824] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:43.206885 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:43.196581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60891 (local address 127.13.223.126:38239)
0928 14:50:43.197088 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:43.197110 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:43.197158 (+ 48us) server_negotiation.cc:407] Connection header received
0928 14:50:43.198146 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:43.198178 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:43.198752 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:43.199214 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:43.199707 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:43.201338 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:43.202433 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:43.203257 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:43.203457 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:43.204487 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:43.204497 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:43.204517 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:43.205106 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:43.205187 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:43.205223 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:43.205322 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:43.205642 (+ 320us) server_negotiation.cc:299] Negotiation successful
0928 14:50:43.205972 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":142,"threads_started":1}
I20260928 14:50:43.207412 3920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1: No bootstrap required, opened a new log
I20260928 14:50:43.210590 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:43.211748 3920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } }
I20260928 14:50:43.212054 3920 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:43.212118 3920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a89c44cff4e45a287ab56ab0d3213a1, State: Initialized, Role: FOLLOWER
I20260928 14:50:43.212620 3920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } }
I20260928 14:50:43.212750 3920 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:43.212838 3920 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:43.212973 3920 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:43.214306 3920 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } }
I20260928 14:50:43.214735 3920 leader_election.cc:302] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a89c44cff4e45a287ab56ab0d3213a1; no voters:
I20260928 14:50:43.215169 3920 leader_election.cc:288] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:43.215327 3930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:43.215610 3930 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [term 1 LEADER]: Becoming Leader. State: Replica: 8a89c44cff4e45a287ab56ab0d3213a1, State: Running, Role: LEADER
I20260928 14:50:43.216076 3930 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } }
I20260928 14:50:43.216907 3920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:43.218119 3931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } } }
I20260928 14:50:43.218257 3931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:43.218631 3932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a89c44cff4e45a287ab56ab0d3213a1. Latest consensus state: current_term: 1 leader_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a89c44cff4e45a287ab56ab0d3213a1" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 38239 } } }
I20260928 14:50:43.218778 3923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:43.213038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59249 (local address 127.13.223.126:38239)
0928 14:50:43.213203 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:43.213210 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:43.213478 (+ 268us) server_negotiation.cc:407] Connection header received
0928 14:50:43.213598 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:43.213601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:43.213657 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:43.213736 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:43.214016 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:43.214830 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:43.216278 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:43.216635 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:43.216771 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:43.218089 (+ 1318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:43.218094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:43.218098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:43.218117 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:43.218158 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:43.218178 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:43.218255 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:43.218487 (+ 232us) server_negotiation.cc:299] Negotiation successful
0928 14:50:43.218595 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 14:50:43.218806 3932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:43.219221 3939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:43.223526 3939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:43.229221 3939 catalog_manager.cc:1347] Generated new cluster ID: 2dc97db3bdb64d28a0844b2795bdde06
I20260928 14:50:43.229439 3939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:43.237088 3939 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:43.238165 3939 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:43.249395 3939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a89c44cff4e45a287ab56ab0d3213a1: Generated new TSK 0
I20260928 14:50:43.250222 3939 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:43.256970 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3849
2026-09-28T14:50:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:43 dist-test-slave-gkhx krb5kdc[3951](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:43 dist-test-slave-gkhx krb5kdc[3951](info): set up 2 sockets
Sep 28 14:50:43 dist-test-slave-gkhx krb5kdc[3951](info): commencing operation
krb5kdc: starting...
W20260928 14:50:44.409097 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.131s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:44 dist-test-slave-gkhx krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607044, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:44.542866 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:36799
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:34451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:36799
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:44.661351 3964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:44.661720 3964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:44.661803 3964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:44.666165 3964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:44.666301 3964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:44.666335 3964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:44.666379 3964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:44.666435 3964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:44.672602 3964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:34451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:36799
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:36799
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:44.673866 3964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:44.675094 3964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:44.686338 3972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:44.686305 3969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:44.686493 3964 server_base.cc:1034] running on GCE node
W20260928 14:50:44.686374 3970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:44.687536 3964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:44.688793 3964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:44.690063 3964 hybrid_clock.cc:648] HybridClock initialized: now 1790607044690040 us; error 39 us; skew 500 ppm
Sep 28 14:50:44 dist-test-slave-gkhx krb5kdc[3951](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607044, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:44.693897 3964 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:44.695156 3964 webserver.cc:460] Webserver started at http://127.13.223.126:37911/ using document root <none> and password file <none>
I20260928 14:50:44.695827 3964 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:44.695931 3964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:44.696202 3964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:44.698685 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "544af6b8bf2d4628976caf41dbd3fd8a"
format_stamp: "Formatted at 2026-09-28 14:50:44 on dist-test-slave-gkhx"
server_key: "562aa4c531a9eac4910d7057e4fdc4f6"
server_key_iv: "028a574f9141bb0483cfac6362137709"
server_key_version: "encryptionkey@0"
I20260928 14:50:44.699349 3964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "544af6b8bf2d4628976caf41dbd3fd8a"
format_stamp: "Formatted at 2026-09-28 14:50:44 on dist-test-slave-gkhx"
server_key: "562aa4c531a9eac4910d7057e4fdc4f6"
server_key_iv: "028a574f9141bb0483cfac6362137709"
server_key_version: "encryptionkey@0"
I20260928 14:50:44.703909 3964 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:44.706732 3979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:44.708223 3964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:44.708384 3964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "544af6b8bf2d4628976caf41dbd3fd8a"
format_stamp: "Formatted at 2026-09-28 14:50:44 on dist-test-slave-gkhx"
server_key: "562aa4c531a9eac4910d7057e4fdc4f6"
server_key_iv: "028a574f9141bb0483cfac6362137709"
server_key_version: "encryptionkey@0"
I20260928 14:50:44.708516 3964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:44.713608 3964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:44.717202 3964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:44.717418 3964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:44.725378 3964 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:36799
I20260928 14:50:44.725405 4032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:36799 every 8 connection(s)
I20260928 14:50:44.726437 3964 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:44.729678 4034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:44.730707 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 3964
I20260928 14:50:44.730823 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:44.731158 14205 external_mini_cluster.cc:1428] Setting key 7c008eef1b83c0eebb275a7dced7eedc
I20260928 14:50:44.737555 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:44.732711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44345 (local address 127.13.223.126:36799)
0928 14:50:44.733235 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:44.733256 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:44.733305 (+ 49us) server_negotiation.cc:407] Connection header received
0928 14:50:44.734210 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:44.734242 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:44.734689 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:44.735115 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:44.735881 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:44.735888 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:44.735905 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:44.735916 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:44.735994 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:44.736024 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:44.736215 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:44.736301 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 14:50:44.736494 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":137,"threads_started":1}
I20260928 14:50:44.737766 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a: Bootstrap starting.
I20260928 14:50:44.740707 4034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:44.740734 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:44.741547 4034 log.cc:824] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:44.744307 4034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a: No bootstrap required, opened a new log
I20260928 14:50:44.744938 4037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:44.743093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46629 (local address 127.13.223.126:36799)
0928 14:50:44.743231 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:44.743236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:44.743463 (+ 227us) server_negotiation.cc:407] Connection header received
0928 14:50:44.743631 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:44.743633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:44.743666 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:44.743738 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:44.744532 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:44.744536 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:44.744538 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 14:50:44.744550 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:44.744578 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 14:50:44.744589 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:44.744671 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:44.744805 (+ 134us) server_negotiation.cc:299] Negotiation successful
0928 14:50:44.744841 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:50:44.748868 4034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } }
I20260928 14:50:44.749239 4034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 14:50:44.749298 4034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544af6b8bf2d4628976caf41dbd3fd8a, State: Initialized, Role: FOLLOWER
I20260928 14:50:44.749789 4034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } }
I20260928 14:50:44.749930 4034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 14:50:44.749980 4034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 14:50:44.750056 4034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:44.751209 4034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } }
I20260928 14:50:44.751566 4034 leader_election.cc:302] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 544af6b8bf2d4628976caf41dbd3fd8a; no voters:
I20260928 14:50:44.751871 4034 leader_election.cc:288] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 14:50:44.752014 4044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:44.752285 4044 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [term 1 LEADER]: Becoming Leader. State: Replica: 544af6b8bf2d4628976caf41dbd3fd8a, State: Running, Role: LEADER
I20260928 14:50:44.752679 4044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } }
I20260928 14:50:44.753309 4034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:44.754478 4046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544af6b8bf2d4628976caf41dbd3fd8a. Latest consensus state: current_term: 1 leader_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } } }
I20260928 14:50:44.754683 4046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:44.755005 4045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "544af6b8bf2d4628976caf41dbd3fd8a" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 36799 } } }
I20260928 14:50:44.755187 4045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:44.755088 4053 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:44.759011 4053 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:44.765851 4053 catalog_manager.cc:1347] Generated new cluster ID: 4bdd16da742a48dc9a10c2af98027507
I20260928 14:50:44.765955 4053 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:44.789309 4053 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:44.789711 4053 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:44.801826 4053 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544af6b8bf2d4628976caf41dbd3fd8a: Generated new TSK 0
I20260928 14:50:44.802841 4053 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 14:50:44.815660 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 3964
2026-09-28T14:50:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1558 ms)
[----------] 18 tests from AuthTokenIssuingTest (27457 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:44 dist-test-slave-gkhx krb5kdc[4066](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:44 dist-test-slave-gkhx krb5kdc[4066](info): set up 2 sockets
Sep 28 14:50:44 dist-test-slave-gkhx krb5kdc[4066](info): commencing operation
krb5kdc: starting...
W20260928 14:50:45.876886 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:45 dist-test-slave-gkhx krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:46.025077 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44337
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:46.154701 4079 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:46.155105 4079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:46.155170 4079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:46.155268 4079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:46.159552 4079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:46.159794 4079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:46.159899 4079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:46.159941 4079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:46.159976 4079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:46.166052 4079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:44337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:46.167379 4079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:46.168489 4079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:46.178543 4085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.178666 4087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.178543 4084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:46.179016 4079 server_base.cc:1034] running on GCE node
I20260928 14:50:46.179739 4079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:46.181211 4079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:46.182456 4079 hybrid_clock.cc:648] HybridClock initialized: now 1790607046182431 us; error 38 us; skew 500 ppm
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:46.191529 4079 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:46.192687 4079 webserver.cc:460] Webserver started at http://127.13.223.126:43899/ using document root <none> and password file <none>
I20260928 14:50:46.193377 4079 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:46.193488 4079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:46.193765 4079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:46.196419 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "39306de486294983beed7d0e72862639"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "bc33c4eee65ec13c1a81f0ee7efb84cd"
server_key_iv: "e68f2454c4a11a34fe99a118c96bb052"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.197125 4079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "39306de486294983beed7d0e72862639"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "bc33c4eee65ec13c1a81f0ee7efb84cd"
server_key_iv: "e68f2454c4a11a34fe99a118c96bb052"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.201792 4079 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 14:50:46.204864 4094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:46.206769 4079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:46.206962 4079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "39306de486294983beed7d0e72862639"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "bc33c4eee65ec13c1a81f0ee7efb84cd"
server_key_iv: "e68f2454c4a11a34fe99a118c96bb052"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.207124 4079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:46.237874 4079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:46.241904 4079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:46.242178 4079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:46.250221 4079 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:44337
I20260928 14:50:46.250528 4146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:44337 every 8 connection(s)
I20260928 14:50:46.251439 4079 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:46.254112 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4079
I20260928 14:50:46.254225 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:46.254532 14205 external_mini_cluster.cc:1428] Setting key 9619eec4cc74eb1630abdac454d1aee7
I20260928 14:50:46.254839 4147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:46.259287 4147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.125@KRBTEST.COM" created.
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:46.281867 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.261899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44337 (local address 127.0.0.1:55328)
0928 14:50:46.262490 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.262584 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.263875 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.264708 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.264718 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.265194 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.265604 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.265629 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.267762 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.267770 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.268847 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.268861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.270509 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.270515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.270683 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.271472 (+ 789us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.271503 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.274839 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.278539 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.278575 (+ 36us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.278596 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.278854 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.279230 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.279233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.279235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.279397 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.280216 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.280222 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.280364 (+ 142us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.280509 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.280942 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":97,"thread_start_us":193,"threads_started":1}
I20260928 14:50:46.281867 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.263004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.13.223.126:44337)
0928 14:50:46.263389 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.263399 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.263449 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:50:46.264218 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.264224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.264426 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.264610 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.265813 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.267580 (+ 1767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.269023 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.269884 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.270143 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.275068 (+ 4925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.275077 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.275096 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.275131 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.278390 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.278986 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.278992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.278994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.279120 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.279577 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.279581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.279582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.280053 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.280258 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.280636 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.280835 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260928 14:50:46.284906 4147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.285797 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.285483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42358)
0928 14:50:46.285628 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.285718 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 14:50:46.286146 4097 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
I20260928 14:50:46.287356 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.287063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42368)
0928 14:50:46.287200 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.287261 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260928 14:50:46.287635 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:36489
--webserver_interface=127.13.223.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:46.289072 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:36489: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
I20260928 14:50:46.338430 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } attempt: 1
I20260928 14:50:46.339495 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.339134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42384)
0928 14:50:46.339319 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.339399 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260928 14:50:46.340416 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.340175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42398)
0928 14:50:46.340291 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.340349 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 14:50:46.340582 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:36489: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
I20260928 14:50:46.408799 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } attempt: 2
I20260928 14:50:46.409739 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.409390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42404)
0928 14:50:46.409569 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.409645 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 14:50:46.410636 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.410297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42420)
0928 14:50:46.410441 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.410522 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:50:46.410825 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:36489: Network error: Client connection negotiation failed: client connection to 127.13.223.125:36489: connect: Connection refused (error 111)
W20260928 14:50:46.413798 4154 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:46.414121 4154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:46.414146 4154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:46.414188 4154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:46.418543 4154 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:46.418704 4154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:46.418725 4154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:46.418742 4154 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:46.418759 4154 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:46.424995 4154 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:36489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:46.426220 4154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:46.427390 4154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:46.437059 4160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.437064 4162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.437063 4159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:46.437443 4154 server_base.cc:1034] running on GCE node
I20260928 14:50:46.438161 4154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:46.439473 4154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:46.440696 4154 hybrid_clock.cc:648] HybridClock initialized: now 1790607046440679 us; error 35 us; skew 500 ppm
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:46.443915 4154 init.cc:377] Logged in from keytab as kudu/127.13.223.125@KRBTEST.COM (short username kudu)
I20260928 14:50:46.445190 4154 webserver.cc:460] Webserver started at http://127.13.223.125:34457/ using document root <none> and password file <none>
I20260928 14:50:46.445976 4154 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:46.446090 4154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:46.446355 4154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:46.448845 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data/instance:
uuid: "8117234c4bb447debb469a4fcd6187d7"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4893127eb0192cbbe7509563c3372d15"
server_key_iv: "d62cb49d9e2d1784d6bc8444987937a0"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.449548 4154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal/instance:
uuid: "8117234c4bb447debb469a4fcd6187d7"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4893127eb0192cbbe7509563c3372d15"
server_key_iv: "d62cb49d9e2d1784d6bc8444987937a0"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.454241 4154 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:46.457248 4170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:46.458899 4154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:46.459138 4154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "8117234c4bb447debb469a4fcd6187d7"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4893127eb0192cbbe7509563c3372d15"
server_key_iv: "d62cb49d9e2d1784d6bc8444987937a0"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.459292 4154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:46.475618 4154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:46.479451 4154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:46.479719 4154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:46.487308 4154 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.125:36489
I20260928 14:50:46.487619 4223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.125:36489 every 8 connection(s)
I20260928 14:50:46.488402 4154 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
I20260928 14:50:46.491429 4224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:46.495914 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4154
I20260928 14:50:46.496037 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-1/wal/instance
I20260928 14:50:46.495795 4224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.496335 14205 external_mini_cluster.cc:1428] Setting key 62b938549a330691cd7abf49e91d073f
I20260928 14:50:46.502002 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } attempt: 3
WARNING: no policy specified for kudu/127.13.223.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.124@KRBTEST.COM" created.
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:46.513990 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.498317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55336 (local address 127.13.223.126:44337)
0928 14:50:46.498490 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.498503 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.499577 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 14:50:46.500570 (+ 993us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.500574 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.500662 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.500771 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.502162 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.503083 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.504654 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.505023 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.505109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.508785 (+ 3676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.508789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.508791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.508821 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.511466 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512119 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.512126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.512131 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.512224 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512686 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.512690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.512692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.512941 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.513052 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.513570 (+ 518us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.513772 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":68}
I20260928 14:50:46.514736 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.502501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42434)
0928 14:50:46.502720 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.502737 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.502810 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.503457 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.503460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.503653 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.503801 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.503809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.505167 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.505170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.505784 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.505796 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.506677 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.506685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.506865 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.507525 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.507545 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.508904 (+ 1359us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.511743 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.511748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.511751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.511995 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.512425 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512429 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.512431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.512526 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.513365 (+ 839us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.513368 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.513490 (+ 122us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.513605 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.514585 (+ 980us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":785}
I20260928 14:50:46.515161 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.498847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44337 (local address 127.0.0.1:55336)
0928 14:50:46.499382 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.499438 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.500349 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.500967 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.500974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.501341 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.501973 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.501995 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.503243 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.503246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.504515 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.504525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.505266 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.505268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.505423 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.506096 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.506115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.508618 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.511594 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.511608 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.511623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.511990 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.512353 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.512362 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.512537 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.513069 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.513079 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.513284 (+ 205us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.513522 (+ 238us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.513928 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":169,"threads_started":1}
I20260928 14:50:46.515165 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.502577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.13.223.125:36489)
0928 14:50:46.503006 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.503018 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.503036 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:46.503068 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.503072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.503272 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.503443 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.503936 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.505058 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.505908 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.506451 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.506594 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.509073 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.509084 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.509105 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.509139 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.511615 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512122 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.512129 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.512131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.512264 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.512676 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.512679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.512681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.513246 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.513404 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.513844 (+ 440us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.514005 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":109,"threads_started":1}
I20260928 14:50:46.517621 4224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.517642 4147 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.518404 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.518084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36872)
0928 14:50:46.518258 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.518323 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 14:50:46.519161 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.518909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36886)
0928 14:50:46.519071 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.519114 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260928 14:50:46.519325 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
I20260928 14:50:46.527004 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:37259
--webserver_interface=127.13.223.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:43291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:46.527557 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.518424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42436)
0928 14:50:46.518647 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.518661 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.518746 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.519309 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.519312 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.519524 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.519648 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.519653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.520508 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.520511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.521165 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.521175 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.521683 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.521686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.521772 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.522373 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.522386 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.523876 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.525738 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.525742 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.525745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.526035 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.526358 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.526361 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.526362 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.526440 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.526949 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.526951 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.527052 (+ 101us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.527177 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.527359 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 14:50:46.527557 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.518468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.13.223.125:36489)
0928 14:50:46.518701 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.518707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.518726 (+ 19us) server_negotiation.cc:407] Connection header received
0928 14:50:46.518832 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.518838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.518916 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.519000 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.519766 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.520376 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.521269 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.521559 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.521667 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.524067 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.524070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.524072 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.524095 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.525610 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.526129 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.526133 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.526134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.526244 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.526531 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.526534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.526536 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.526842 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.526950 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.527225 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.527426 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":57}
I20260928 14:50:46.528517 4224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.529335 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.529022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36902)
0928 14:50:46.529179 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.529270 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260928 14:50:46.530026 4171 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
I20260928 14:50:46.531279 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.530994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36912)
0928 14:50:46.531165 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.531216 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 14:50:46.531607 4224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
I20260928 14:50:46.550603 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 1
I20260928 14:50:46.551630 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.551233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36922)
0928 14:50:46.551427 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.551508 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 14:50:46.552623 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.552330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36932)
0928 14:50:46.552465 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.552535 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260928 14:50:46.552841 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
I20260928 14:50:46.580976 4224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 1
I20260928 14:50:46.582093 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.581687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36948)
0928 14:50:46.581900 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.581983 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 14:50:46.583288 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.582978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36964)
0928 14:50:46.583137 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.583203 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260928 14:50:46.583494 4224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
I20260928 14:50:46.628089 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 2
I20260928 14:50:46.629024 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.628680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36978)
0928 14:50:46.628857 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.628929 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 14:50:46.630024 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.629744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36986)
0928 14:50:46.629881 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.629957 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260928 14:50:46.630246 4147 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
W20260928 14:50:46.651194 4230 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:46.651583 4230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:46.651610 4230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:46.651687 4230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:50:46.651722 4224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 2
I20260928 14:50:46.652753 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.652395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:36996)
0928 14:50:46.652588 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.652660 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 14:50:46.653630 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.653381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:37008)
0928 14:50:46.653500 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.653557 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 14:50:46.653883 4224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:37259: Network error: Client connection negotiation failed: client connection to 127.13.223.124:37259: connect: Connection refused (error 111)
W20260928 14:50:46.656687 4230 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:46.656795 4230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:46.656833 4230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:46.656893 4230 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:46.656942 4230 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:46.662889 4230 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:43291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:44337,127.13.223.125:36489,127.13.223.124:37259
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:37259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:46.664167 4230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:46.665267 4230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:46.674916 4236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.675079 4239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:46.674916 4237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:46.675230 4230 server_base.cc:1034] running on GCE node
I20260928 14:50:46.675967 4230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:46.677273 4230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:46.678521 4230 hybrid_clock.cc:648] HybridClock initialized: now 1790607046678507 us; error 42 us; skew 500 ppm
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:46.681950 4230 init.cc:377] Logged in from keytab as kudu/127.13.223.124@KRBTEST.COM (short username kudu)
I20260928 14:50:46.683156 4230 webserver.cc:460] Webserver started at http://127.13.223.124:33607/ using document root <none> and password file <none>
I20260928 14:50:46.683825 4230 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:46.683931 4230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:46.684206 4230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:46.686766 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data/instance:
uuid: "bd0ff4a4313244b69739dcdb714a586d"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4cbfea2e8ad14dde2b3e0141f3ae1cc0"
server_key_iv: "d142d80e98ffd2fdd93a2691e4d61321"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.687440 4230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal/instance:
uuid: "bd0ff4a4313244b69739dcdb714a586d"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4cbfea2e8ad14dde2b3e0141f3ae1cc0"
server_key_iv: "d142d80e98ffd2fdd93a2691e4d61321"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.691831 4230 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 14:50:46.694877 4246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:46.696426 4230 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:46.696566 4230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "bd0ff4a4313244b69739dcdb714a586d"
format_stamp: "Formatted at 2026-09-28 14:50:46 on dist-test-slave-gkhx"
server_key: "4cbfea2e8ad14dde2b3e0141f3ae1cc0"
server_key_iv: "d142d80e98ffd2fdd93a2691e4d61321"
server_key_version: "encryptionkey@0"
I20260928 14:50:46.696671 4230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:46.708567 4230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:46.712502 4230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:46.712759 4230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:46.721060 4230 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.124:37259
I20260928 14:50:46.721359 4298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.124:37259 every 8 connection(s)
I20260928 14:50:46.722259 4230 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
I20260928 14:50:46.725404 4299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:46.725772 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4230
I20260928 14:50:46.725905 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790606739990168-14205-0/minicluster-data/master-2/wal/instance
I20260928 14:50:46.726231 14205 external_mini_cluster.cc:1428] Setting key 6695c004a0fb67f401142b6bd98436ea
I20260928 14:50:46.729456 4147 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 3
I20260928 14:50:46.731127 4299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:46.742992 4150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.729952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:37034)
0928 14:50:46.730227 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.730242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.730321 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.731168 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.731172 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.731363 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.731474 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.731483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.732920 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.732924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.733741 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.733754 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.734869 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.734872 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.734969 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.735573 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.735594 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.737092 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.740324 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.740328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.740330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.740647 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.741070 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.741073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.741075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.741248 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.742353 (+ 1105us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.742357 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.742470 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.742621 (+ 151us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.742790 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 14:50:46.743803 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.727759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37022 (local address 127.13.223.124:37259)
0928 14:50:46.728256 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.728274 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.728313 (+ 39us) server_negotiation.cc:407] Connection header received
0928 14:50:46.729093 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.729115 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.729596 (+ 481us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.729987 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.730985 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.732717 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.733834 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.734740 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.734925 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.737935 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.737938 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.737941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.737971 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.740164 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.740743 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.740750 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.740758 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.740946 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.741293 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.741297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.741299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.741760 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.741981 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.742313 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.742697 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":180,"server-negotiator.queue_time_us":323,"spinlock_wait_cycles":6400,"thread_start_us":160,"threads_started":1}
I20260928 14:50:46.743801 4304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.730467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37034 (local address 127.13.223.124:37259)
0928 14:50:46.730853 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.730862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.730873 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:46.730933 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.730936 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.730980 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.731091 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.731613 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.732717 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.733864 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.734740 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.734926 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.737290 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.737301 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.737311 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.737363 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.740173 (+ 2810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.740756 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.740759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.740760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.740946 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.741671 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.741674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.741676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.742220 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.743291 (+ 1071us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.743364 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.743488 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":273,"server-negotiator.queue_time_us":329,"spinlock_wait_cycles":5632,"thread_start_us":216,"threads_started":1}
I20260928 14:50:46.744944 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.733205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:44337 (local address 127.0.0.1:55350)
0928 14:50:46.734085 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.734129 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.734285 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.734623 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.734632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.735167 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.735321 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.735327 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.736379 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.736385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.737493 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.737508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.738066 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.738069 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.738153 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.738725 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.738745 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.740900 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.743035 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.743043 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.743049 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.743321 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.743610 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.743613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.743615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.743707 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.744327 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.744336 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.744461 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.744617 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.744787 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":106,"threads_started":1}
I20260928 14:50:46.744944 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.733242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.13.223.126:44337)
0928 14:50:46.733357 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.733361 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.734218 (+ 857us) server_negotiation.cc:407] Connection header received
0928 14:50:46.734392 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.734394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.734445 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.734537 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.735441 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.736197 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.737640 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.737943 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.738052 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.741051 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.741055 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.741057 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.741078 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.742927 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.743409 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.743412 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.743414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.743496 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.743838 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.743842 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.743844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.744139 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.744266 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.744699 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.744815 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:50:46.745190 4224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } attempt: 3
I20260928 14:50:46.747040 4299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.748805 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: Bootstrap starting.
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:50:46.752439 4147 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: Neither blocks nor log segments found. Creating new log.
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:46.753470 4147 log.cc:824] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: Log is configured to *not* fsync() on all Append() calls
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:46.756057 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.745654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:37040)
0928 14:50:46.745944 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.745959 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.746035 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.746310 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.746313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.746505 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.746688 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.746700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.748011 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.748015 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.748817 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.748831 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.749403 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.749407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.749487 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.750049 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.750065 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.751637 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.753761 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.753765 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.753767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.754031 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.754603 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.754607 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.754609 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.754738 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.755394 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.755397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.755508 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.755699 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.755854 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191}
I20260928 14:50:46.756896 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.745919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.13.223.124:37259)
0928 14:50:46.746024 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.746028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.746041 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:50:46.746116 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.746119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.746162 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.746813 (+ 651us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.746901 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.747644 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.748968 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.749274 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.749764 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.751785 (+ 2021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.751789 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.751791 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.751853 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.753632 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.754320 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.754323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.754324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.754442 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.754958 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.754962 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.754964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.755233 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.755341 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.756304 (+ 963us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.756684 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42,"spinlock_wait_cycles":62336}
I20260928 14:50:46.756728 4147 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: No bootstrap required, opened a new log
I20260928 14:50:46.759334 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.746849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55358 (local address 127.13.223.126:44337)
0928 14:50:46.746954 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.746958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.747145 (+ 187us) server_negotiation.cc:407] Connection header received
0928 14:50:46.747319 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.747325 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.747648 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.747709 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.748621 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.749392 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.750606 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.750935 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.751010 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.755311 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.755315 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.755317 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.755345 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.757137 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.757706 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.757710 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.757711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.757799 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.758197 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.758201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.758202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.758406 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.758595 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.759037 (+ 442us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.759170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:50:46.760358 4224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: Bootstrap starting.
I20260928 14:50:46.761281 4147 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.761482 4147 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39306de486294983beed7d0e72862639, State: Initialized, Role: FOLLOWER
I20260928 14:50:46.761739 4147 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.762171 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.762389 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.763057 4147 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:46.763209 4224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:46.763824 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.747555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42444 (local address 127.13.223.125:36489)
0928 14:50:46.747668 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.747674 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.747888 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:50:46.747963 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.747967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.748013 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.748094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.748674 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.749397 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.750573 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.750936 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.751010 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.753898 (+ 2888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.753903 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.753908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.753936 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.755777 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.756816 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.756819 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.756821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.756906 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.762628 (+ 5722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.762633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.762634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.762873 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.762955 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.763549 (+ 594us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.763677 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":52}
I20260928 14:50:46.764138 4224 log.cc:824] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:46.764932 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.747497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:36489 (local address 127.0.0.1:42444)
0928 14:50:46.747805 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.747819 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.747892 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.748257 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.748262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.748423 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.748530 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.748540 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.749606 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.749610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.750403 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.750413 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.751163 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.751166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.751275 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.751832 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.751849 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.753726 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.755908 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.755914 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.755919 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.756695 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.757278 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.757280 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.757282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.757380 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.763072 (+ 5692us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.763079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.763323 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.764550 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.764737 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239}
I20260928 14:50:46.766008 4299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:46.767808 4224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: No bootstrap required, opened a new log
W20260928 14:50:46.769073 4320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:46.769164 4320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:46.772310 4224 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.772581 4224 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8117234c4bb447debb469a4fcd6187d7, State: Initialized, Role: FOLLOWER
I20260928 14:50:46.768658 4079 master_runner.cc:186] Error getting master registration for 127.13.223.125:36489: OK, instance_id { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" instance_seqno: 1790607046485183 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:46.772800 4079 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:46.772943 4224 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.774541 4224 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:46.775439 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.760748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.13.223.125:36489)
0928 14:50:46.763979 (+ 3231us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.763984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.763998 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:50:46.764036 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.764038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.764078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.764179 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.765104 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.765859 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.767632 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.767935 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.768033 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.771240 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.771244 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.771247 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.771275 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.773053 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.773588 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.773593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.773608 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.773695 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.774016 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.774019 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.774021 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.774264 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.774439 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.775141 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.775264 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3166,"thread_start_us":106,"threads_started":1}
Sep 28 14:50:46 dist-test-slave-gkhx krb5kdc[4066](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607046, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:50:46.779824 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:46.781320 4120 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "39306de486294983beed7d0e72862639"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55358
I20260928 14:50:46.781581 4120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:50:46.781698 4120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:46.782882 4305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.767432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:37259 (local address 127.0.0.1:37042)
0928 14:50:46.773167 (+ 5735us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:46.773184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:46.773277 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:46.773765 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:46.773770 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:46.773972 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:46.774087 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.774107 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.775381 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.775384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.776158 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:46.776170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.777078 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.777081 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:46.777171 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.777715 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:46.777730 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:46.779265 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:46.781283 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.781285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.781288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.781519 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.781806 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:46.781810 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:46.781815 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:46.781892 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:46.782336 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:46.782339 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:46.782424 (+ 85us) client_negotiation.cc:769] Sending connection context
0928 14:50:46.782519 (+ 95us) client_negotiation.cc:240] Negotiation successful
0928 14:50:46.782693 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5606}
I20260928 14:50:46.783257 4120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.783336 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.767725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37042 (local address 127.13.223.124:37259)
0928 14:50:46.768324 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.768331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.773363 (+ 5032us) server_negotiation.cc:407] Connection header received
0928 14:50:46.773396 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.773401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.773445 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.773509 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.774214 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.774989 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.776558 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.776870 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.776967 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.779404 (+ 2437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.779408 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.779410 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.779434 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.781163 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.781628 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.781631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.781632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.781717 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.781980 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.781983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.781985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.782216 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.782304 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.783068 (+ 764us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.783208 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260928 14:50:46.784818 4120 leader_election.cc:288] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [CANDIDATE]: Term 1 election: Requested vote from peers 8117234c4bb447debb469a4fcd6187d7 (127.13.223.125:36489), bd0ff4a4313244b69739dcdb714a586d (127.13.223.124:37259)
I20260928 14:50:46.785379 4198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39306de486294983beed7d0e72862639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8117234c4bb447debb469a4fcd6187d7"
I20260928 14:50:46.785617 4198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:46.787187 4198 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39306de486294983beed7d0e72862639 in term 1.
I20260928 14:50:46.787307 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.787449 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.787719 4097 leader_election.cc:302] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39306de486294983beed7d0e72862639, 8117234c4bb447debb469a4fcd6187d7; no voters:
I20260928 14:50:46.788107 4308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:46.788340 4308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [term 1 LEADER]: Becoming Leader. State: Replica: 39306de486294983beed7d0e72862639, State: Running, Role: LEADER
I20260928 14:50:46.788717 4308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.790131 4299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: Bootstrap starting.
I20260928 14:50:46.790688 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39306de486294983beed7d0e72862639. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.790822 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: This master's current role is: LEADER
W20260928 14:50:46.791083 4333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:46.791162 4333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:46.792141 4344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:46.794008 4344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:46.794392 4299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:46.795502 4299 log.cc:824] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:46.783365 4154 master_runner.cc:186] Error getting master registration for 127.13.223.124:37259: OK, instance_id { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" instance_seqno: 1790607046718831 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:46.796255 4154 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:46.798487 4299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: No bootstrap required, opened a new log
I20260928 14:50:46.803613 4299 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.804663 4198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [term 1 FOLLOWER]: Refusing update from remote peer 39306de486294983beed7d0e72862639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:46.804867 4299 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd0ff4a4313244b69739dcdb714a586d, State: Initialized, Role: FOLLOWER
I20260928 14:50:46.805363 4299 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } }
I20260928 14:50:46.805600 4308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:46.808060 4299 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:46.810441 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39306de486294983beed7d0e72862639. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.810628 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.810786 4272 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39306de486294983beed7d0e72862639" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bd0ff4a4313244b69739dcdb714a586d"
I20260928 14:50:46.811237 4273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:46.814836 4273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [term 1 FOLLOWER]: Refusing update from remote peer 39306de486294983beed7d0e72862639: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:46.815709 4308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:46.819607 4272 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 39306de486294983beed7d0e72862639 in term 1.
I20260928 14:50:46.826313 4347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.826649 4347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.830824 4347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39306de486294983beed7d0e72862639. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.831003 4347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.832585 4303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.791114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37046 (local address 127.13.223.124:37259)
0928 14:50:46.791433 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.791438 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.808993 (+ 17555us) server_negotiation.cc:407] Connection header received
0928 14:50:46.810525 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.810529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.810623 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.810680 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.816262 (+ 5582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.817083 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.818419 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.818829 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.818914 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.826456 (+ 7542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.826463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.826468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.826501 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.828320 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.828958 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.828961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.828963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.829059 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.830379 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.830383 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.830384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.830705 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.830831 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.832239 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.832403 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241}
W20260928 14:50:46.833448 4361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260928 14:50:46.836532 4346 mvcc.cc:205] Tried to move back new op lower bound from 7334326463698882560 to 7334326463650344960. Current Snapshot: MvccSnapshot[applied={T|T < 7334326463698882560}]
W20260928 14:50:46.836771 4361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:46.837095 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.837244 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:46.838626 4322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.838783 4322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.839879 4348 mvcc.cc:205] Tried to move back new op lower bound from 7334326463698882560 to 7334326463650344960. Current Snapshot: MvccSnapshot[applied={T|T < 7334326463698882560}]
I20260928 14:50:46.847162 4341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.847383 4341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639 [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:46.849532 4151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.791182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55370 (local address 127.13.223.126:44337)
0928 14:50:46.791315 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.791320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.808590 (+ 17270us) server_negotiation.cc:407] Connection header received
0928 14:50:46.811903 (+ 3313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.811907 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.811960 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.812011 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.823005 (+ 10994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.823869 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.825116 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.825468 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.825550 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.833289 (+ 7739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.833295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.833298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.833332 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.835252 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.838671 (+ 3419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.838679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.838681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.838792 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.847363 (+ 8571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.847368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.847369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.847666 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.847817 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.848985 (+ 1168us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.849280 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:50:46.849505 4347 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "39306de486294983beed7d0e72862639" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "39306de486294983beed7d0e72862639" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 44337 } } peers { permanent_uuid: "8117234c4bb447debb469a4fcd6187d7" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 36489 } } peers { permanent_uuid: "bd0ff4a4313244b69739dcdb714a586d" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 37259 } } }
I20260928 14:50:46.849665 4347 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:46.850732 4228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:46.790663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42468 (local address 127.13.223.125:36489)
0928 14:50:46.790791 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.790796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.808797 (+ 18001us) server_negotiation.cc:407] Connection header received
0928 14:50:46.809308 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.809311 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.809364 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.809416 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.819600 (+ 10184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.820475 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.821943 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:46.822314 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:46.822392 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:46.835936 (+ 13544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:46.835941 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:46.835944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:46.835983 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:46.837780 (+ 1797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.846727 (+ 8947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.846734 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.846736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.846912 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:46.848195 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:46.848199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:46.848201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:46.848459 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:46.848591 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:46.850380 (+ 1789us) server_negotiation.cc:299] Negotiation successful
0928 14:50:46.850524 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 14:50:46.853178 4344 catalog_manager.cc:1347] Generated new cluster ID: 99ded824334d4bc3becfad68596b3687
I20260928 14:50:46.853272 4344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:46.865509 4363 mvcc.cc:205] Tried to move back new op lower bound from 7334326463698882560 to 7334326463650344960. Current Snapshot: MvccSnapshot[applied={T|T < 7334326463698882560}]
I20260928 14:50:46.910907 4344 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:46.912561 4344 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:46.937383 4344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39306de486294983beed7d0e72862639: Generated new TSK 0
I20260928 14:50:46.938508 4344 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:50:46.976979 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.972896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.13.223.125:36489)
0928 14:50:46.973065 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.973071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.975467 (+ 2396us) server_negotiation.cc:407] Connection header received
0928 14:50:46.975788 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.975794 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.975872 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.975945 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.976793 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42476: BlockingRecv error: recv got EOF from 127.0.0.1:42476 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260928 14:50:46.979056 4151 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.973996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55372 (local address 127.13.223.126:44337)
0928 14:50:46.974135 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.974141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.977839 (+ 3698us) server_negotiation.cc:407] Connection header received
0928 14:50:46.978085 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.978089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.978150 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.978214 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.978901 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55372: BlockingRecv error: recv got EOF from 127.0.0.1:55372 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260928 14:50:46.979954 4303 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.974809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.13.223.124:37259)
0928 14:50:46.974961 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.974966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.978858 (+ 3892us) server_negotiation.cc:407] Connection header received
0928 14:50:46.979064 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.979068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.979123 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.979178 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.979819 (+ 641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37060: BlockingRecv error: recv got EOF from 127.0.0.1:37060 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 14:50:46.985641 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.983605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.13.223.125:36489)
0928 14:50:46.983775 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.983781 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.984427 (+ 646us) server_negotiation.cc:407] Connection header received
0928 14:50:46.984698 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.984701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.984789 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.984846 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.985486 (+ 640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42486: BlockingRecv error: recv got EOF from 127.0.0.1:42486 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 14:50:46.992727 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.990189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42502 (local address 127.13.223.125:36489)
0928 14:50:46.990338 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.990345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.991200 (+ 855us) server_negotiation.cc:407] Connection header received
0928 14:50:46.991537 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.991541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.991628 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.991691 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.992586 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42502: BlockingRecv error: recv got EOF from 127.0.0.1:42502 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 14:50:46.999761 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:46.997649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.13.223.125:36489)
0928 14:50:46.997783 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:46.997788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:46.998473 (+ 685us) server_negotiation.cc:407] Connection header received
0928 14:50:46.998759 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:46.998763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:46.998826 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:46.998900 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:46.999609 (+ 709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42514: BlockingRecv error: recv got EOF from 127.0.0.1:42514 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:50:47.008816 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.006722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42520 (local address 127.13.223.125:36489)
0928 14:50:47.006904 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.006909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.007610 (+ 701us) server_negotiation.cc:407] Connection header received
0928 14:50:47.007857 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.007861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.007922 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.007982 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.008676 (+ 694us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42520: BlockingRecv error: recv got EOF from 127.0.0.1:42520 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260928 14:50:47.021435 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.020002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.13.223.125:36489)
0928 14:50:47.020171 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.020178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.020386 (+ 208us) server_negotiation.cc:407] Connection header received
0928 14:50:47.020554 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.020557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.020636 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.020723 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.021302 (+ 579us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42528: BlockingRecv error: recv got EOF from 127.0.0.1:42528 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:50:47.041652 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.040093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42544 (local address 127.13.223.125:36489)
0928 14:50:47.040264 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.040269 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.040569 (+ 300us) server_negotiation.cc:407] Connection header received
0928 14:50:47.040718 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.040721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.040777 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.040860 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.041455 (+ 595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42544: BlockingRecv error: recv got EOF from 127.0.0.1:42544 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 14:50:47.077509 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.075991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.13.223.125:36489)
0928 14:50:47.076208 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.076213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.076420 (+ 207us) server_negotiation.cc:407] Connection header received
0928 14:50:47.076640 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.076644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.076719 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.076814 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.077384 (+ 570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42556: BlockingRecv error: recv got EOF from 127.0.0.1:42556 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
W20260928 14:50:47.145383 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.144012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.13.223.125:36489)
0928 14:50:47.144259 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.144264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.144403 (+ 139us) server_negotiation.cc:407] Connection header received
0928 14:50:47.144593 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.144596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.144647 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.144743 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.145231 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42560: BlockingRecv error: recv got EOF from 127.0.0.1:42560 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260928 14:50:47.277374 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.276016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42566 (local address 127.13.223.125:36489)
0928 14:50:47.276253 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.276257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.276351 (+ 94us) server_negotiation.cc:407] Connection header received
0928 14:50:47.276496 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.276498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.276550 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.276625 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.277243 (+ 618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42566: BlockingRecv error: recv got EOF from 127.0.0.1:42566 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260928 14:50:47.537441 4228 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:47.536203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.13.223.125:36489)
0928 14:50:47.536407 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:47.536411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:47.536428 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:50:47.536506 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:47.536509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:47.536556 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:47.536637 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:47.537309 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42580: BlockingRecv error: recv got EOF from 127.0.0.1:42580 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 14:50:47.792618 4333 catalog_manager.cc:1259] Loaded cluster ID: 99ded824334d4bc3becfad68596b3687
I20260928 14:50:47.792719 4333 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: loading cluster ID for follower catalog manager: success
I20260928 14:50:47.794414 4333 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: acquiring CA information for follower catalog manager: success
I20260928 14:50:47.795873 4333 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8117234c4bb447debb469a4fcd6187d7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:47.838303 4361 catalog_manager.cc:1259] Loaded cluster ID: 99ded824334d4bc3becfad68596b3687
I20260928 14:50:47.838435 4361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: loading cluster ID for follower catalog manager: success
I20260928 14:50:47.840282 4361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: acquiring CA information for follower catalog manager: success
I20260928 14:50:47.841614 4361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bd0ff4a4313244b69739dcdb714a586d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:48.056444 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:48.051891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42590 (local address 127.13.223.125:36489)
0928 14:50:48.052252 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:48.052256 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:48.052274 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:48.052332 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:48.052335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:48.052518 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:48.052632 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:48.052916 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.053566 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.054612 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.054890 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.054991 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:48.055475 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:48.055992 (+ 517us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:48.056099 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:48.056218 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 14:50:48.056292 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":103,"threads_started":1}
I20260928 14:50:48.065196 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:48.060178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.13.223.124:37259)
0928 14:50:48.060536 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:48.060539 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:48.060556 (+ 17us) server_negotiation.cc:407] Connection header received
0928 14:50:48.060659 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:48.060661 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:48.060782 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:48.060901 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:48.061275 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.061930 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.063192 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.063605 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.063735 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:48.064098 (+ 363us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:48.064677 (+ 579us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:48.064787 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:48.064889 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 14:50:48.064955 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260928 14:50:48.074307 4434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:48.069873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42596 (local address 127.13.223.125:36489)
0928 14:50:48.070009 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:48.070013 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:48.070091 (+ 78us) server_negotiation.cc:407] Connection header received
0928 14:50:48.070260 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:48.070263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:48.070322 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:48.070372 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:48.070821 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.071761 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.072941 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.073307 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.073384 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:48.073755 (+ 371us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:48.073849 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:48.073929 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:48.074041 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 14:50:48.074126 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 14:50:48.074628 4447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:48.069289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55384 (local address 127.13.223.126:44337)
0928 14:50:48.069573 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:48.069580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:48.069602 (+ 22us) server_negotiation.cc:407] Connection header received
0928 14:50:48.069861 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:48.069865 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:48.070063 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:48.070197 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:48.070496 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.071496 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.072785 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.073152 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.073284 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:48.073653 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:48.074076 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:48.074166 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:48.074340 (+ 174us) server_negotiation.cc:299] Negotiation successful
0928 14:50:48.074412 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":103,"threads_started":1}
I20260928 14:50:48.075279 4440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:48.069771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37086 (local address 127.13.223.124:37259)
0928 14:50:48.069911 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:48.069916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:48.070278 (+ 362us) server_negotiation.cc:407] Connection header received
0928 14:50:48.070464 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:48.070470 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:48.070535 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:48.070636 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:48.070980 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.071511 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.073805 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:48.074159 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:48.074261 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:48.074683 (+ 422us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:48.074777 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:48.074826 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:48.075062 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 14:50:48.075158 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 14:50:48.078764 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4079
I20260928 14:50:48.085762 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4154
I20260928 14:50:48.092448 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4230
2026-09-28T14:50:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3278 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 14:50:48 dist-test-slave-gkhx krb5kdc[4451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 14:50:48 dist-test-slave-gkhx krb5kdc[4451](info): set up 2 sockets
Sep 28 14:50:48 dist-test-slave-gkhx krb5kdc[4451](info): commencing operation
krb5kdc: starting...
W20260928 14:50:49.169122 14205 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/test-user.keytab.
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T14:50:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T14:50:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.13.223.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
Entry for principal kudu/127.13.223.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab.
I20260928 14:50:49.307165 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46179
--webserver_interface=127.13.223.126
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 14:50:49.425747 4464 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:49.426182 4464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:49.426261 4464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:49.426347 4464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:49.431039 4464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:49.431182 4464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:49.431222 4464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:49.431265 4464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:49.431300 4464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:49.437384 4464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.126.keytab
--principal=kudu/127.13.223.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.126:46179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.13.223.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:49.438704 4464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:49.439831 4464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:49.449811 4469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.449774 4470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.449783 4472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:49.450044 4464 server_base.cc:1034] running on GCE node
I20260928 14:50:49.450886 4464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:49.452152 4464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:49.453369 4464 hybrid_clock.cc:648] HybridClock initialized: now 1790607049453337 us; error 60 us; skew 500 ppm
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:49.456667 4464 init.cc:377] Logged in from keytab as kudu/127.13.223.126@KRBTEST.COM (short username kudu)
I20260928 14:50:49.457803 4464 webserver.cc:460] Webserver started at http://127.13.223.126:45923/ using document root <none> and password file <none>
I20260928 14:50:49.458442 4464 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:49.458549 4464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:49.458814 4464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:49.461138 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data/instance:
uuid: "870e40d077df47c699e72630e1f9b60f"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2d676d1f24daf5aafaec9442359ad821"
server_key_iv: "32af7ce65080643c4d2c3d59c07bc565"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.461782 4464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance:
uuid: "870e40d077df47c699e72630e1f9b60f"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2d676d1f24daf5aafaec9442359ad821"
server_key_iv: "32af7ce65080643c4d2c3d59c07bc565"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.466146 4464 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:49.469210 4479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:49.471057 4464 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:49.471246 4464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal
uuid: "870e40d077df47c699e72630e1f9b60f"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2d676d1f24daf5aafaec9442359ad821"
server_key_iv: "32af7ce65080643c4d2c3d59c07bc565"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.471400 4464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:49.483168 4464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:49.487555 4464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:49.487795 4464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:49.495679 4464 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.126:46179
I20260928 14:50:49.496016 4533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.126:46179 every 8 connection(s)
I20260928 14:50:49.496799 4464 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/data/info.pb
I20260928 14:50:49.500061 4534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:49.504943 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4464
I20260928 14:50:49.505077 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-0/wal/instance
I20260928 14:50:49.505496 14205 external_mini_cluster.cc:1428] Setting key 074d47350ef0df80d0c6be681fb0f20b
I20260928 14:50:49.506042 4534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.125@KRBTEST.COM" created.
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:49.528667 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.509330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46179 (local address 127.0.0.1:53152)
0928 14:50:49.510342 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.510404 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.511797 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.512854 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.512862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.513391 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.513834 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.513861 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.515923 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.515930 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.517344 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.517362 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.518510 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.518516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.518683 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.519474 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.519514 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.521941 (+ 2427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:49.524827 (+ 2886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.524839 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.524855 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.525260 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.525712 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.525716 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.525718 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.525889 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.526899 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:49.526909 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:49.527085 (+ 176us) client_negotiation.cc:769] Sending connection context
0928 14:50:49.527304 (+ 219us) client_negotiation.cc:240] Negotiation successful
0928 14:50:49.527739 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":795,"mutex_wait_us":25,"thread_start_us":176,"threads_started":1}
I20260928 14:50:49.528673 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.509330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53152 (local address 127.13.223.126:46179)
0928 14:50:49.510049 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.510072 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.510616 (+ 544us) server_negotiation.cc:407] Connection header received
0928 14:50:49.512236 (+ 1620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.512248 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.512581 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.512810 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.514048 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.515722 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.517538 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.518078 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.518267 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.522169 (+ 3902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:49.522181 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:49.522201 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:49.522239 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:49.524654 (+ 2415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.525403 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.525412 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.525416 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.525586 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.526001 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.526006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.526008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.526713 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:49.526882 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:49.527407 (+ 525us) server_negotiation.cc:299] Negotiation successful
0928 14:50:49.527878 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":135,"server-negotiator.queue_time_us":546,"spinlock_wait_cycles":58112,"thread_start_us":203,"threads_started":1}
I20260928 14:50:49.531466 4534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:49.532359 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.532069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55046)
0928 14:50:49.532202 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.532289 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 14:50:49.532646 4483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
I20260928 14:50:49.533792 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.533446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55062)
0928 14:50:49.533618 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.533686 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260928 14:50:49.534111 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:42359: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
Entry for principal kudu/127.13.223.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab.
I20260928 14:50:49.539076 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:42359
--webserver_interface=127.13.223.125
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:49.583436 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } attempt: 1
I20260928 14:50:49.584692 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.584231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55068)
0928 14:50:49.584514 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.584589 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176}
I20260928 14:50:49.585654 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.585341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55078)
0928 14:50:49.585487 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.585562 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:50:49.585888 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:42359: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
I20260928 14:50:49.654079 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } attempt: 2
I20260928 14:50:49.655040 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.654735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55094)
0928 14:50:49.654894 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.654960 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 14:50:49.655833 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.655550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55108)
0928 14:50:49.655667 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.655753 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 14:50:49.655952 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.125:42359: Network error: Client connection negotiation failed: client connection to 127.13.223.125:42359: connect: Connection refused (error 111)
W20260928 14:50:49.662889 4541 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:49.663280 4541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:49.663341 4541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:49.663424 4541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 14:50:49.667779 4541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:49.667882 4541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:49.667920 4541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:49.667958 4541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:49.667997 4541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:49.673915 4541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.125.keytab
--principal=kudu/127.13.223.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.125:42359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.13.223.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:49.675264 4541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:49.676405 4541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 14:50:49.685933 4546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.686008 4549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.685930 4547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:49.686213 4541 server_base.cc:1034] running on GCE node
I20260928 14:50:49.687052 4541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:49.688290 4541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:49.689505 4541 hybrid_clock.cc:648] HybridClock initialized: now 1790607049689476 us; error 59 us; skew 500 ppm
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:49.692723 4541 init.cc:377] Logged in from keytab as kudu/127.13.223.125@KRBTEST.COM (short username kudu)
I20260928 14:50:49.693840 4541 webserver.cc:460] Webserver started at http://127.13.223.125:46831/ using document root <none> and password file <none>
I20260928 14:50:49.694486 4541 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:49.694613 4541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:49.694873 4541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:49.697167 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data/instance:
uuid: "e13105e2947146498563881f3bc60176"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2bc3a1ff36f4c869ef24614a3d57dec2"
server_key_iv: "0a65e39be15ea7b75eb5ad4a44c7801d"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.697817 4541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal/instance:
uuid: "e13105e2947146498563881f3bc60176"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2bc3a1ff36f4c869ef24614a3d57dec2"
server_key_iv: "0a65e39be15ea7b75eb5ad4a44c7801d"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.702241 4541 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 14:50:49.705415 4556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:49.706920 4541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:49.707110 4541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal
uuid: "e13105e2947146498563881f3bc60176"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "2bc3a1ff36f4c869ef24614a3d57dec2"
server_key_iv: "0a65e39be15ea7b75eb5ad4a44c7801d"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.707248 4541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:49.722617 4541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:49.726199 4541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:49.726426 4541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:49.733928 4541 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.125:42359
I20260928 14:50:49.734261 4608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.125:42359 every 8 connection(s)
I20260928 14:50:49.735155 4541 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/data/info.pb
I20260928 14:50:49.737461 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4541
I20260928 14:50:49.737598 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-1/wal/instance
I20260928 14:50:49.737903 14205 external_mini_cluster.cc:1428] Setting key 01e98bd51cdee243c50e4b60177df4e8
I20260928 14:50:49.738515 4609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:49.743182 4609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.13.223.124@KRBTEST.COM; defaulting to no policy
I20260928 14:50:49.747195 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.13.223.124@KRBTEST.COM" created.
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:49.760265 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.745860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.13.223.126:46179)
0928 14:50:49.746038 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.746043 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.747193 (+ 1150us) server_negotiation.cc:407] Connection header received
0928 14:50:49.748416 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.748418 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.748488 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.748603 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.749804 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.750626 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.751973 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.752868 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.752994 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.756171 (+ 3177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:49.756175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:49.756177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:49.756204 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:49.758010 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758496 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.758499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.758501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.758615 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.759005 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.759008 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.759010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.759252 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:49.759360 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:49.759936 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 14:50:49.760069 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:50:49.760398 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.747669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55118)
0928 14:50:49.747802 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.747819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.747943 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.748827 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.748829 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.749043 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.749216 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.749230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.750614 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.750620 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.751482 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.751494 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.752479 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.752484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.752620 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.753223 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.753238 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.754974 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:49.757578 (+ 2604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.757581 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.757584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.758057 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.758538 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.758571 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.758652 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.759541 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:49.759545 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:49.759648 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 14:50:49.759791 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 14:50:49.760037 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 14:50:49.761382 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.746339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46179 (local address 127.0.0.1:53160)
0928 14:50:49.746957 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.747013 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.748191 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.748671 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.748682 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.749165 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.749595 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.749609 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.750767 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.750770 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.751850 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.751861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.753079 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.753085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.753222 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.753763 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.753788 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.755997 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:49.758120 (+ 2123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758131 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.758143 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.758379 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.758716 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.758721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.758879 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.759396 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:49.759411 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:49.759665 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 14:50:49.759871 (+ 206us) client_negotiation.cc:240] Negotiation successful
0928 14:50:49.760363 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":112,"thread_start_us":221,"threads_started":1}
I20260928 14:50:49.761384 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.747782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55118 (local address 127.13.223.125:42359)
0928 14:50:49.748157 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.748169 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.748216 (+ 47us) server_negotiation.cc:407] Connection header received
0928 14:50:49.748374 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.748393 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.748671 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.748855 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.749365 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.750477 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.751625 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.752328 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.752526 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.755162 (+ 2636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:49.755169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:49.755184 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:49.755217 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:49.757436 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758212 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.758219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.758224 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.758390 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.758755 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.758758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.758760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.759398 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:49.759517 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:49.759908 (+ 391us) server_negotiation.cc:299] Negotiation successful
0928 14:50:49.760244 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":268,"thread_start_us":154,"threads_started":1}
I20260928 14:50:49.763705 4534 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:49.763717 4609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:49.764786 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.764303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51384)
0928 14:50:49.764656 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.764716 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260928 14:50:49.765597 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.765246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51386)
0928 14:50:49.765428 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.765526 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260928 14:50:49.765769 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
Entry for principal kudu/127.13.223.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab.
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:49.770838 14205 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
/tmp/dist-test-tasktMPbyY/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:38277
--webserver_interface=127.13.223.124
--webserver_port=0
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.13.223.84:39291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 14:50:49.774251 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.764524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.13.223.125:42359)
0928 14:50:49.764687 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.764691 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.765088 (+ 397us) server_negotiation.cc:407] Connection header received
0928 14:50:49.765255 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.765258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.765331 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.765417 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.765931 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.766683 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.767721 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.767958 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.768100 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.770526 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:49.770529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:49.770531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:49.770608 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:49.772204 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.772683 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.772688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.772691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.772787 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:49.773079 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:49.773082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:49.773083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:49.773396 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:49.773527 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:49.773960 (+ 433us) server_negotiation.cc:299] Negotiation successful
0928 14:50:49.774113 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":90}
I20260928 14:50:49.774401 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.764302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55126)
0928 14:50:49.765002 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.765014 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.765146 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.765434 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.765436 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.765679 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.765829 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.765839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.766792 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.766795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.767575 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.767585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.768137 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.768140 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.768230 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.768837 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.768851 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.770341 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:49.772347 (+ 2006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.772350 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.772351 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.772577 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.772892 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:49.772895 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:49.772896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:49.772982 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:49.773539 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:49.773541 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:49.773678 (+ 137us) client_negotiation.cc:769] Sending connection context
0928 14:50:49.773828 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 14:50:49.774049 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636}
I20260928 14:50:49.775494 4609 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:49.776484 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.776098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51396)
0928 14:50:49.776267 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.776386 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 14:50:49.777089 4557 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
I20260928 14:50:49.778117 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.777804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51406)
0928 14:50:49.777957 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.778024 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 14:50:49.778319 4609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
I20260928 14:50:49.797111 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 1
I20260928 14:50:49.798149 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.797761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51422)
0928 14:50:49.797961 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.798044 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260928 14:50:49.799113 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.798833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51426)
0928 14:50:49.798963 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.799027 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260928 14:50:49.799300 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
I20260928 14:50:49.827662 4609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 1
I20260928 14:50:49.828754 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.828318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51442)
0928 14:50:49.828501 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.828633 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 14:50:49.829869 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.829601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51456)
0928 14:50:49.829732 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.829795 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 14:50:49.830036 4609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
I20260928 14:50:49.874516 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 2
I20260928 14:50:49.875533 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.875174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51470)
0928 14:50:49.875347 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.875431 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260928 14:50:49.876493 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.876242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51486)
0928 14:50:49.876361 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.876422 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 14:50:49.876672 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
W20260928 14:50:49.896849 4615 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 14:50:49.897197 4615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 14:50:49.897222 4615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 14:50:49.897264 4615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 14:50:49.898250 4609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 2
I20260928 14:50:49.899215 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.898854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51500)
0928 14:50:49.899036 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.899116 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260928 14:50:49.900005 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.899772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51506)
0928 14:50:49.899887 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.899941 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 14:50:49.900202 4609 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
W20260928 14:50:49.902648 4615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 14:50:49.902736 4615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 14:50:49.902783 4615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 14:50:49.902822 4615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 14:50:49.902858 4615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 14:50:49.909073 4615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.13.223.84:39291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.13.223.126:46179,127.13.223.125:42359,127.13.223.124:38277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/krb5kdc/kudu_127.13.223.124.keytab
--principal=kudu/127.13.223.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.13.223.124:38277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.13.223.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 13:20:11 UTC on ccycloud.quasar-gzyzyf.root.comops.site
build id 4677202
I20260928 14:50:49.910341 4615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 14:50:49.911566 4615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 14:50:49.921451 4615 server_base.cc:1034] running on GCE node
W20260928 14:50:49.921393 4622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.921391 4623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 14:50:49.921393 4625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 14:50:49.922348 4615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 14:50:49.923612 4615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 14:50:49.924824 4615 hybrid_clock.cc:648] HybridClock initialized: now 1790607049924790 us; error 39 us; skew 500 ppm
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 14:50:49.927873 4615 init.cc:377] Logged in from keytab as kudu/127.13.223.124@KRBTEST.COM (short username kudu)
I20260928 14:50:49.928865 4615 webserver.cc:460] Webserver started at http://127.13.223.124:33315/ using document root <none> and password file <none>
I20260928 14:50:49.929432 4615 fs_manager.cc:359] Metadata directory not provided
I20260928 14:50:49.929498 4615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 14:50:49.929678 4615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 14:50:49.931993 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data/instance:
uuid: "1826fabc92934c1a8077263ab63a1327"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "da8443d8ad462544a01185b9c6749b87"
server_key_iv: "ef0b460ac277da2e39c091e39ff8b40e"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.932564 4615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal/instance:
uuid: "1826fabc92934c1a8077263ab63a1327"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "da8443d8ad462544a01185b9c6749b87"
server_key_iv: "ef0b460ac277da2e39c091e39ff8b40e"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.936849 4615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 14:50:49.939785 4632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 14:50:49.941176 4615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 14:50:49.941298 4615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data,/tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal
uuid: "1826fabc92934c1a8077263ab63a1327"
format_stamp: "Formatted at 2026-09-28 14:50:49 on dist-test-slave-gkhx"
server_key: "da8443d8ad462544a01185b9c6749b87"
server_key_iv: "ef0b460ac277da2e39c091e39ff8b40e"
server_key_version: "encryptionkey@0"
I20260928 14:50:49.941403 4615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 14:50:49.966869 4615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 14:50:49.970505 4615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 14:50:49.970789 4615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 14:50:49.975836 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 3
I20260928 14:50:49.976946 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.976551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51518)
0928 14:50:49.976722 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.976815 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 14:50:49.977847 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.977535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51530)
0928 14:50:49.977672 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.977747 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260928 14:50:49.978109 4534 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.13.223.124:38277: Network error: Client connection negotiation failed: client connection to 127.13.223.124:38277: connect: Connection refused (error 111)
I20260928 14:50:49.979722 4615 rpc_server.cc:307] RPC server started. Bound to: 127.13.223.124:38277
I20260928 14:50:49.980065 4684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.13.223.124:38277 every 8 connection(s)
I20260928 14:50:49.980862 4615 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/data/info.pb
I20260928 14:50:49.983924 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 14:50:49.988109 4685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:49.989813 14205 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu as pid 4615
I20260928 14:50:49.989998 14205 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasktMPbyY/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790606739990168-14205-0/minicluster-data/master-2/wal/instance
I20260928 14:50:49.990352 14205 external_mini_cluster.cc:1428] Setting key f0ae69f2876c0f6e8a3baf93ec5eb1ad
I20260928 14:50:49.991379 4609 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 3
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.125@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
Sep 28 14:50:49 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:50:50.004457 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.990255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53170 (local address 127.13.223.126:46179)
0928 14:50:49.990428 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.990434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.991672 (+ 1238us) server_negotiation.cc:407] Connection header received
0928 14:50:49.992924 (+ 1252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.992928 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.993003 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.993109 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.994311 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.994925 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.996232 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.996615 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.996721 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.000283 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.000289 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.000294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.000334 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.002208 (+ 1874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.002892 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.002896 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.002898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.002998 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.003328 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.003334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.003337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.003555 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.003667 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.004136 (+ 469us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.004272 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 14:50:50.004478 4612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.991814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51540)
0928 14:50:49.991943 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.991957 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.992064 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.993285 (+ 1221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.993288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.993520 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.993641 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.993649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.995562 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.995567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.996372 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.996385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.997075 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.997081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.997185 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.997823 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.997845 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.999374 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:50.002004 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.002008 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.002010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.002236 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.002767 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.002771 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.002773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.002884 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.003834 (+ 950us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:50.003838 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:50.003962 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 14:50:50.004105 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 14:50:50.004285 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 14:50:50.005653 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.990826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.126:46179 (local address 127.0.0.1:53170)
0928 14:50:49.991472 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:49.991564 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:49.992701 (+ 1137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:49.993162 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:49.993172 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:49.993798 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:49.994109 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.994124 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.995049 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.995055 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.996088 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:49.996099 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.996781 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.996785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:49.996966 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.997685 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:49.997703 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:49.999963 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:50.002348 (+ 2385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.002355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.002367 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.002775 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.003126 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.003129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.003131 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.003212 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.003667 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:50.003676 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:50.003855 (+ 179us) client_negotiation.cc:769] Sending connection context
0928 14:50:50.004062 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 14:50:50.004414 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":3,"spinlock_wait_cycles":4864,"thread_start_us":167,"threads_started":1}
I20260928 14:50:50.005659 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.991182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51536 (local address 127.13.223.124:38277)
0928 14:50:49.991677 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.991689 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.992027 (+ 338us) server_negotiation.cc:407] Connection header received
0928 14:50:49.992720 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.992747 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.993076 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.993265 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.994234 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.995423 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.996366 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.996947 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.997042 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.000729 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.000758 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.000763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.000794 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.003280 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.003766 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.003769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.003772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.003848 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.004145 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.004149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.004150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.004349 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.004527 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.004909 (+ 382us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.005042 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":276,"server-negotiator.queue_time_us":385,"spinlock_wait_cycles":4352,"thread_start_us":167,"threads_started":1}
I20260928 14:50:50.005659 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:49.992262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.13.223.124:38277)
0928 14:50:49.992567 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:49.992574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:49.992585 (+ 11us) server_negotiation.cc:407] Connection header received
0928 14:50:49.992726 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:49.992747 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:49.993076 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:49.993756 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:49.993855 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.995423 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.996512 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:49.996934 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:49.997186 (+ 252us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:49.999606 (+ 2420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:49.999615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:49.999634 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:49.999670 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.001862 (+ 2192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.002359 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.002365 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.002367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.002623 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.003006 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.003009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.003011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.003696 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.003865 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.004870 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.005134 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":225,"server-negotiator.queue_time_us":245,"spinlock_wait_cycles":12288,"thread_start_us":124,"threads_started":1}
I20260928 14:50:50.007169 4685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:50.011327 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: Bootstrap starting.
Sep 28 14:50:50 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:50.014572 4609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: Neither blocks nor log segments found. Creating new log.
Sep 28 14:50:50 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.126@KRBTEST.COM
I20260928 14:50:50.015607 4609 log.cc:824] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:50.017958 4609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: No bootstrap required, opened a new log
I20260928 14:50:50.018025 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.007870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55128 (local address 127.13.223.125:42359)
0928 14:50:50.008011 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.008016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.008032 (+ 16us) server_negotiation.cc:407] Connection header received
0928 14:50:50.008076 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.008079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.008129 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.008228 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.008802 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.009535 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.010682 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.010927 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.011074 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.013638 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.013645 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.013651 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.013691 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.015570 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.016516 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.016522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.016524 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.016625 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.016978 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.016985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.016988 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.017221 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.017342 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.017682 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.017815 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 14:50:50.018515 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.007702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.125:42359 (local address 127.0.0.1:55128)
0928 14:50:50.007826 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:50.007841 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:50.007925 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:50.008319 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:50.008322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:50.008533 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:50.008662 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.008670 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.009680 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.009684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.010505 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.010516 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.011071 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.011075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.011184 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.011763 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:50.011780 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:50.013449 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:50.016129 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.016134 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.016136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.016372 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.016751 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.016755 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.016758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.016860 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.017341 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:50.017344 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:50.017464 (+ 120us) client_negotiation.cc:769] Sending connection context
0928 14:50:50.018163 (+ 699us) client_negotiation.cc:240] Negotiation successful
0928 14:50:50.018344 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260928 14:50:50.019496 4685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } has no permanent_uuid. Determining permanent_uuid...
I20260928 14:50:50.020095 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.008684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.13.223.126:46179)
0928 14:50:50.008797 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.008802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.008917 (+ 115us) server_negotiation.cc:407] Connection header received
0928 14:50:50.009167 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.009170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.009222 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.009308 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.009949 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.010599 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.012258 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.012516 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.012634 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.015746 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.015749 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.015752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.015785 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.018142 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.018754 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.018758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.018760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.018853 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.019187 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.019192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.019197 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.019409 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.019666 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.019797 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.019927 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 14:50:50.023339 4609 raft_consensus.cc:348] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.023643 4609 raft_consensus.cc:729] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e13105e2947146498563881f3bc60176, State: Initialized, Role: FOLLOWER
I20260928 14:50:50.024081 4609 consensus_queue.cc:260] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.024715 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.024986 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.025758 4609 sys_catalog.cc:564] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: configured and running, proceeding with master startup.
Sep 28 14:50:50 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.124@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
Sep 28 14:50:50 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.13.223.125@KRBTEST.COM
I20260928 14:50:50.030517 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.020235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.13.223.124:38277)
0928 14:50:50.020358 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.020362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.020720 (+ 358us) server_negotiation.cc:407] Connection header received
0928 14:50:50.020833 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.020837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.020876 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.020919 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.021488 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.022330 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.023359 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.023714 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.023820 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.026794 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.026802 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.026808 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.026846 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.028666 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.029148 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.029154 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.029156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.029249 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.029561 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.029566 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.029568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.029806 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.029914 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.030168 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.030360 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":36}
I20260928 14:50:50.030517 4687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.020002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51550)
0928 14:50:50.020604 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:50.020633 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:50.020733 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:50.021044 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:50.021047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:50.021234 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:50.021328 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.021335 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.022462 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.022466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.023246 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.023257 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.023826 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.023830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.023919 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.024814 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:50.024829 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:50.026533 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:50.028792 (+ 2259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.028796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.028797 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.029055 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.029359 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.029364 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.029366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.029460 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.029913 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:50.029915 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:50.030011 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 14:50:50.030125 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 14:50:50.030309 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"mutex_wait_us":3}
W20260928 14:50:50.032295 4703 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:50.032394 4703 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:50.033504 4685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: Bootstrap starting.
I20260928 14:50:50.030205 4541 master_runner.cc:186] Error getting master registration for 127.13.223.126:46179: OK, instance_id { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" instance_seqno: 1790607049493450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:50.034318 4541 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:50.036160 4685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:50.036986 4685 log.cc:824] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:50.037942 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.021329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55142 (local address 127.13.223.125:42359)
0928 14:50:50.021464 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.021471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.021484 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:50:50.021631 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.021633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.021683 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.021762 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.022650 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.023146 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.024221 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.024557 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.024666 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.028045 (+ 3379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.028050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.028053 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.028115 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.035211 (+ 7096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.035917 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.035922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.035924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.036039 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.036382 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.036386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.036388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.036660 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.036870 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.037629 (+ 759us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.037766 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 14:50:50.039558 4685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: No bootstrap required, opened a new log
I20260928 14:50:50.042037 14205 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 14:50:50.043429 4685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.043854 4685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1826fabc92934c1a8077263ab63a1327, State: Initialized, Role: FOLLOWER
I20260928 14:50:50.044277 4685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.043485 4507 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "870e40d077df47c699e72630e1f9b60f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53186
I20260928 14:50:50.045063 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.045261 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.046039 4685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 14:50:50.053112 4717 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 14:50:50.053310 4717 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 14:50:50.051043 4615 master_runner.cc:186] Error getting master registration for 127.13.223.126:46179: OK, instance_id { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" instance_seqno: 1790607049493450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 14:50:50.054450 4615 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:50.142290 4534 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } attempt: 4
Sep 28 14:50:50 dist-test-slave-gkhx krb5kdc[4451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790607049, etypes {rep=17 tkt=17 ses=17}, kudu/127.13.223.126@KRBTEST.COM for kudu/127.13.223.124@KRBTEST.COM
I20260928 14:50:50.158356 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.143583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.13.223.124:38277)
0928 14:50:50.143743 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.143748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.143768 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:50.143821 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.143824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.143874 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.143969 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.144490 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.145286 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.146377 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.146795 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.146903 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.154503 (+ 7600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.154508 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.154512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.154598 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.156390 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.156978 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.156982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.156984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.157077 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.157380 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.157383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.157385 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.157633 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.157733 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.158040 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.158180 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 14:50:50.158418 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.142929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.13.223.124:38277 (local address 127.0.0.1:51556)
0928 14:50:50.143294 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 14:50:50.143311 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 14:50:50.143419 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 14:50:50.143995 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 14:50:50.143998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 14:50:50.144235 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0928 14:50:50.144353 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.144360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.145412 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.145416 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.146255 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 14:50:50.146267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.146912 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.146915 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 14:50:50.147004 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.152469 (+ 5465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 14:50:50.152491 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 14:50:50.154283 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 14:50:50.156564 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.156569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.156571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.156872 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.157178 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 14:50:50.157181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 14:50:50.157183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 14:50:50.157285 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 14:50:50.157738 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 14:50:50.157741 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 14:50:50.157857 (+ 116us) client_negotiation.cc:769] Sending connection context
0928 14:50:50.157976 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 14:50:50.158200 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263}
I20260928 14:50:50.162474 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f: Bootstrap starting.
I20260928 14:50:50.165776 4534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f: Neither blocks nor log segments found. Creating new log.
I20260928 14:50:50.166769 4534 log.cc:824] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f: Log is configured to *not* fsync() on all Append() calls
I20260928 14:50:50.169425 4534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f: No bootstrap required, opened a new log
I20260928 14:50:50.172941 4534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.173218 4534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870e40d077df47c699e72630e1f9b60f, State: Initialized, Role: FOLLOWER
I20260928 14:50:50.173614 4534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.174270 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.174598 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.175288 4534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 14:50:50.175383 4507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 14:50:50.175540 4507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:50.176976 4507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.178032 4507 leader_election.cc:288] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [CANDIDATE]: Term 1 election: Requested vote from peers e13105e2947146498563881f3bc60176 (127.13.223.125:42359), 1826fabc92934c1a8077263ab63a1327 (127.13.223.124:38277)
I20260928 14:50:50.178979 4659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "870e40d077df47c699e72630e1f9b60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1826fabc92934c1a8077263ab63a1327"
I20260928 14:50:50.179265 4659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:50.179443 4583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "870e40d077df47c699e72630e1f9b60f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e13105e2947146498563881f3bc60176"
I20260928 14:50:50.179672 4583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [term 0 FOLLOWER]: Advancing to term 1
I20260928 14:50:50.180861 4659 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870e40d077df47c699e72630e1f9b60f in term 1.
I20260928 14:50:50.181016 4583 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870e40d077df47c699e72630e1f9b60f in term 1.
I20260928 14:50:50.184577 4480 leader_election.cc:302] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1826fabc92934c1a8077263ab63a1327, 870e40d077df47c699e72630e1f9b60f; no voters:
I20260928 14:50:50.184970 4720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 14:50:50.185225 4720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [term 1 LEADER]: Becoming Leader. State: Replica: 870e40d077df47c699e72630e1f9b60f, State: Running, Role: LEADER
I20260928 14:50:50.185616 4720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } }
I20260928 14:50:50.187697 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870e40d077df47c699e72630e1f9b60f. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.187811 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:50.190840 4741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 14:50:50.191275 4464 master_runner.cc:263] No leader master found from master 870e40d077df47c699e72630e1f9b60f
I20260928 14:50:50.191344 4464 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 14:50:50.194392 4741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 14:50:50.199662 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.185988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.13.223.126:46179)
0928 14:50:50.186213 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.186218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.188369 (+ 2151us) server_negotiation.cc:407] Connection header received
0928 14:50:50.188833 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.188837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.188889 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.188944 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.191310 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.192056 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.193246 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.193579 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.193688 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.195793 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.195798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.195801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.195830 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.197664 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.198174 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.198178 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.198180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.198269 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.198611 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.198614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.198616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.198836 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.198961 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.199342 (+ 381us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.199502 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260928 14:50:50.207119 4613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.184262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55150 (local address 127.13.223.125:42359)
0928 14:50:50.184378 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.184382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.194830 (+ 10448us) server_negotiation.cc:407] Connection header received
0928 14:50:50.195756 (+ 926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.195760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.195823 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.195882 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.199196 (+ 3314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.199944 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.201501 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.201856 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.201960 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.203615 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.203619 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.203622 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.203651 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.205186 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.205705 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.205709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.205711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.205798 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.206079 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.206082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.206083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.206307 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.206438 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.206816 (+ 378us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.206948 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:50:50.208640 4690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:50.183621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.13.223.124:38277)
0928 14:50:50.183743 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.183748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.188753 (+ 5005us) server_negotiation.cc:407] Connection header received
0928 14:50:50.190152 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.190154 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.190200 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.190249 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.193270 (+ 3021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.194000 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.198323 (+ 4323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:50.198701 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:50.198789 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:50.203230 (+ 4441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 14:50:50.203234 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 14:50:50.203237 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 14:50:50.203271 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 14:50:50.205794 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.206747 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.206751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.206753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.206852 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 14:50:50.207154 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 14:50:50.207159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 14:50:50.207160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 14:50:50.207380 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 14:50:50.207519 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:50.208345 (+ 826us) server_negotiation.cc:299] Negotiation successful
0928 14:50:50.208486 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 14:50:50.211879 4583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [term 1 FOLLOWER]: Refusing update from remote peer 870e40d077df47c699e72630e1f9b60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:50.211879 4659 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [term 1 FOLLOWER]: Refusing update from remote peer 870e40d077df47c699e72630e1f9b60f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 14:50:50.212709 4720 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:50.212978 4739 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 14:50:50.219077 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870e40d077df47c699e72630e1f9b60f. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.219242 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: This master's current role is: FOLLOWER
W20260928 14:50:50.222379 4638 tablet.cc:2369] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 14:50:50.224263 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 870e40d077df47c699e72630e1f9b60f. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.224421 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.225911 4720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.226058 4720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:50.228276 4692 sys_catalog.cc:455] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.228423 4692 sys_catalog.cc:458] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.228646 4745 mvcc.cc:205] Tried to move back new op lower bound from 7334326477638025216 to 7334326477564243968. Current Snapshot: MvccSnapshot[applied={T|T < 7334326477638025216}]
I20260928 14:50:50.232252 4743 mvcc.cc:205] Tried to move back new op lower bound from 7334326477638025216 to 7334326477564243968. Current Snapshot: MvccSnapshot[applied={T|T < 7334326477638025216}]
I20260928 14:50:50.234406 4741 catalog_manager.cc:1347] Generated new cluster ID: 7cecf41586174f44b43e9e9d99ba64b8
I20260928 14:50:50.234491 4741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 14:50:50.234665 4744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.234803 4744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f [sys.catalog]: This master's current role is: LEADER
I20260928 14:50:50.238116 4706 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "870e40d077df47c699e72630e1f9b60f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870e40d077df47c699e72630e1f9b60f" member_type: VOTER last_known_addr { host: "127.13.223.126" port: 46179 } } peers { permanent_uuid: "e13105e2947146498563881f3bc60176" member_type: VOTER last_known_addr { host: "127.13.223.125" port: 42359 } } peers { permanent_uuid: "1826fabc92934c1a8077263ab63a1327" member_type: VOTER last_known_addr { host: "127.13.223.124" port: 38277 } } }
I20260928 14:50:50.238265 4706 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 14:50:50.242698 4747 mvcc.cc:205] Tried to move back new op lower bound from 7334326477638025216 to 7334326477564243968. Current Snapshot: MvccSnapshot[applied={T|T < 7334326477638025216}]
I20260928 14:50:50.251462 4741 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 14:50:50.252838 4741 catalog_manager.cc:1504] Loading token signing keys...
I20260928 14:50:50.283468 4741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 870e40d077df47c699e72630e1f9b60f: Generated new TSK 0
I20260928 14:50:50.284503 4741 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 14:50:50.321339 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.319850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.13.223.125:42359)
0928 14:50:50.320266 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.320273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.320291 (+ 18us) server_negotiation.cc:407] Connection header received
0928 14:50:50.320354 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.320361 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.320428 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.320557 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.321189 (+ 632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55162: BlockingRecv error: recv got EOF from 127.0.0.1:55162 (error 108)
Metrics: {"server-negotiator.queue_time_us":303}
W20260928 14:50:50.322878 4538 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.319688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53206 (local address 127.13.223.126:46179)
0928 14:50:50.320100 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.320108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.321765 (+ 1657us) server_negotiation.cc:407] Connection header received
0928 14:50:50.321825 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.321832 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.321911 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.322003 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.322703 (+ 700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53206: BlockingRecv error: recv got EOF from 127.0.0.1:53206 (error 108)
Metrics: {"server-negotiator.queue_time_us":215}
W20260928 14:50:50.324004 4690 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.321567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.13.223.124:38277)
0928 14:50:50.321705 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.321712 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.321727 (+ 15us) server_negotiation.cc:407] Connection header received
0928 14:50:50.322148 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.322153 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.322214 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.322301 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.323818 (+ 1517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51574: BlockingRecv error: recv got EOF from 127.0.0.1:51574 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 14:50:50.329008 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.327368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.13.223.125:42359)
0928 14:50:50.327523 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.327528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.327713 (+ 185us) server_negotiation.cc:407] Connection header received
0928 14:50:50.327947 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.327953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.328055 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.328148 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.328835 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55166: BlockingRecv error: recv got EOF from 127.0.0.1:55166 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 14:50:50.333590 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.332682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55168 (local address 127.13.223.125:42359)
0928 14:50:50.332805 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.332810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.332830 (+ 20us) server_negotiation.cc:407] Connection header received
0928 14:50:50.332876 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.332878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.332919 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.333027 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.333469 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55168: BlockingRecv error: recv got EOF from 127.0.0.1:55168 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 14:50:50.338464 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.337228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.13.223.125:42359)
0928 14:50:50.337350 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.337353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.337573 (+ 220us) server_negotiation.cc:407] Connection header received
0928 14:50:50.337727 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.337730 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.337785 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.337871 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.338341 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55180: BlockingRecv error: recv got EOF from 127.0.0.1:55180 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 14:50:50.346128 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.344704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55182 (local address 127.13.223.125:42359)
0928 14:50:50.344914 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.344920 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.345130 (+ 210us) server_negotiation.cc:407] Connection header received
0928 14:50:50.345330 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.345334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.345406 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.345507 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.345986 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55182: BlockingRecv error: recv got EOF from 127.0.0.1:55182 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
W20260928 14:50:50.357941 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.356563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55192 (local address 127.13.223.125:42359)
0928 14:50:50.356735 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.356743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.356921 (+ 178us) server_negotiation.cc:407] Connection header received
0928 14:50:50.357078 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.357081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.357135 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.357215 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.357776 (+ 561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55192: BlockingRecv error: recv got EOF from 127.0.0.1:55192 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260928 14:50:50.377921 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.376429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55200 (local address 127.13.223.125:42359)
0928 14:50:50.376627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.376633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.376847 (+ 214us) server_negotiation.cc:407] Connection header received
0928 14:50:50.376992 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.376994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.377046 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.377121 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.377789 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55200: BlockingRecv error: recv got EOF from 127.0.0.1:55200 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260928 14:50:50.414433 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.413059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55206 (local address 127.13.223.125:42359)
0928 14:50:50.413221 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.413229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.413425 (+ 196us) server_negotiation.cc:407] Connection header received
0928 14:50:50.413576 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.413579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.413629 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.413696 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.414293 (+ 597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55206: BlockingRecv error: recv got EOF from 127.0.0.1:55206 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260928 14:50:50.482153 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.480765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55220 (local address 127.13.223.125:42359)
0928 14:50:50.480912 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.480917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.481164 (+ 247us) server_negotiation.cc:407] Connection header received
0928 14:50:50.481355 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.481358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.481409 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.481514 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.482044 (+ 530us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55220: BlockingRecv error: recv got EOF from 127.0.0.1:55220 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 14:50:50.613941 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.612713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55234 (local address 127.13.223.125:42359)
0928 14:50:50.612880 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.612887 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.613007 (+ 120us) server_negotiation.cc:407] Connection header received
0928 14:50:50.613153 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.613156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.613212 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.613295 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.613793 (+ 498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55234: BlockingRecv error: recv got EOF from 127.0.0.1:55234 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260928 14:50:50.873463 4613 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 14:50:50.872377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37540 (local address 127.13.223.125:42359)
0928 14:50:50.872544 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:50.872548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:50.872612 (+ 64us) server_negotiation.cc:407] Connection header received
0928 14:50:50.872731 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:50.872734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:50.872783 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:50.872869 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 14:50:50.873338 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37540: BlockingRecv error: recv got EOF from 127.0.0.1:37540 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 14:50:51.033793 4703 catalog_manager.cc:1259] Loaded cluster ID: 7cecf41586174f44b43e9e9d99ba64b8
I20260928 14:50:51.033886 4703 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: loading cluster ID for follower catalog manager: success
I20260928 14:50:51.035552 4703 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: acquiring CA information for follower catalog manager: success
I20260928 14:50:51.036700 4703 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e13105e2947146498563881f3bc60176: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:51.054857 4717 catalog_manager.cc:1259] Loaded cluster ID: 7cecf41586174f44b43e9e9d99ba64b8
I20260928 14:50:51.054946 4717 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: loading cluster ID for follower catalog manager: success
I20260928 14:50:51.056764 4717 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: acquiring CA information for follower catalog manager: success
I20260928 14:50:51.058027 4717 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1826fabc92934c1a8077263ab63a1327: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 14:50:51.392746 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:51.387916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.13.223.125:42359)
0928 14:50:51.388172 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:51.388176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:51.388348 (+ 172us) server_negotiation.cc:407] Connection header received
0928 14:50:51.388452 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:51.388455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:51.388589 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:51.388704 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:51.389140 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.389767 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.390874 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.391146 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.391247 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:51.391710 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:51.392233 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:51.392348 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:51.392531 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 14:50:51.392600 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":81,"threads_started":1}
I20260928 14:50:51.401330 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:51.396753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56300 (local address 127.13.223.124:38277)
0928 14:50:51.397013 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:51.397017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:51.397031 (+ 14us) server_negotiation.cc:407] Connection header received
0928 14:50:51.397127 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:51.397129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:51.397257 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:51.397385 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:51.397695 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.398396 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.399578 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.399824 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.399918 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:51.400310 (+ 392us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:51.400830 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:51.400927 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:51.401100 (+ 173us) server_negotiation.cc:299] Negotiation successful
0928 14:50:51.401171 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":97,"threads_started":1}
I20260928 14:50:51.410571 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:51.405449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37560 (local address 127.13.223.125:42359)
0928 14:50:51.405606 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:51.405612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:51.405918 (+ 306us) server_negotiation.cc:407] Connection header received
0928 14:50:51.406137 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:51.406140 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:51.406212 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:51.406266 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:51.406843 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.407537 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.409254 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.409523 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.409655 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:51.409948 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:51.410044 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:51.410117 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:51.410294 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 14:50:51.410391 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260928 14:50:51.410528 4832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:51.405391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.13.223.126:46179)
0928 14:50:51.406141 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:51.406146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:51.406159 (+ 13us) server_negotiation.cc:407] Connection header received
0928 14:50:51.406263 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:51.406266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:51.406450 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:51.406629 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:51.406881 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.407787 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.408688 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.408939 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.409060 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:51.409441 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:51.409840 (+ 399us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:51.409927 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:51.410070 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 14:50:51.410345 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":90,"threads_started":1}
I20260928 14:50:51.410694 4825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 14:50:51.405964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56310 (local address 127.13.223.124:38277)
0928 14:50:51.406078 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 14:50:51.406085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 14:50:51.406135 (+ 50us) server_negotiation.cc:407] Connection header received
0928 14:50:51.406292 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 14:50:51.406295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 14:50:51.406353 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 14:50:51.406436 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 14:50:51.406756 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.407623 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.409220 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 14:50:51.409551 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 14:50:51.409645 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 14:50:51.410048 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 14:50:51.410146 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 14:50:51.410222 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 14:50:51.410449 (+ 227us) server_negotiation.cc:299] Negotiation successful
0928 14:50:51.410511 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 14:50:51.420938 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4464
I20260928 14:50:51.428280 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4541
I20260928 14:50:51.435657 14205 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasktMPbyY/build/debug/bin/kudu with pid 4615
2026-09-28T14:50:51Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3343 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6622 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (311438 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST